Video frame adjusting method and device, electronic equipment and storage medium

Abstract

The method comprises: determining, by a quantization neural network, a first optical flow from a first video frame to a second video frame and a second optical flow from the second video frame to the first video frame based on the first video frame and the second video frame in an initial video; determining a third optical flow from an intermediate frame to the first video frame and a fourth optical flow from the intermediate frame to the second video frame based on the first optical flow and the second optical flow; determining the intermediate frame according to the two initial video frames, the third optical flow and the fourth optical flow; and inserting the intermediate frame between the two initial video frames. The method provided by the embodiments of the present disclosure achieves the purpose of inserting frames between two adjacent video frames, while ensuring the playing effect of the high frame rate video after the insertion.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of video, and particularly relates to a video frame adjusting method and device, an electronic device and a storage medium. BACKGROUND

[0002] Video is an important medium for information dissemination on the Internet. Factors such as color, frame rate and definition of video can affect the playing effect of the video, and further affect the viewing experience of users. Among them, the higher the frame rate of the video is, the smoother the video plays, and the better the viewing experience of the user is.

[0003] With the development of video playing device hardware, more and more video playing devices support the playing of high frame rate video.

[0004] However, the video frame rate improvement effect achieved by the current video frame rate improvement technology is not ideal. SUMMARY

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a video frame adjusting method, device, electronic device and storage medium, which achieve the purpose of inserting frames between two adjacent video frames, and at the same time can guarantee the playing effect of the high frame rate video after frame insertion.

[0006] In a first aspect, the embodiments of the present disclosure provide a video frame adjusting method, which comprises:

[0007] Based on a first video frame and a second video frame in an initial video, a first optical flow from the first video frame to the second video frame and a second optical flow from the second video frame to the first video frame are determined by a quantization neural network, the first video frame and the second video frame being two adjacent initial video frames;

[0008] Based on the first optical flow and the second optical flow, a third optical flow from an intermediate frame to the first video frame and a fourth optical flow from the intermediate frame to the second video frame are determined, wherein the intermediate frame is an estimated video frame to be inserted between the first video frame and the second video frame;

[0009] The intermediate frame is determined according to the first video frame, the second video frame, the third optical flow and the fourth optical flow;

[0010] In the initial video, the intermediate frame is inserted between the first video frame and the second video frame to obtain a target video.

[0011] In a second aspect, the embodiments of the present disclosure also provide a video frame adjusting device, which comprises:

[0012] The first determining module is configured to determine, based on a first video frame and a second video frame in an initial video, a first optical flow from the first video frame to the second video frame and a second optical flow from the second video frame to the first video frame by using a quantization neural network, the first video frame and the second video frame being two adjacent initial video frames.

[0013] The second determining module is configured to determine, based on the first optical flow and the second optical flow, a third optical flow from an intermediate frame to the first video frame and a fourth optical flow from the intermediate frame to the second video frame, the intermediate frame being an estimated video frame to be inserted between the first video frame and the second video frame.

[0014] The third determining module is configured to determine the intermediate frame according to the first video frame, the second video frame, the third optical flow, and the fourth optical flow.

[0015] The frame insertion module is configured to insert the intermediate frame between the first video frame and the second video frame in the initial video to obtain a target video.

[0016] In a third aspect, an electronic device is provided, and the electronic device includes:

[0017] one or more processors;

[0018] a storage device configured to store one or more programs;

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the video frame adjustment method as described above.

[0020] In a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the video frame adjustment method as described above.

[0021] In a fifth aspect, a computer program product is provided, and the computer program product includes a computer program or instructions, which, when executed by a processor, implements the video frame adjustment method as described above.

[0022] The technical solutions provided in the embodiments of the present disclosure have at least the following advantages:

[0023] The video frame adjustment method provided in the embodiments of the present disclosure is based on a first video frame and a second video frame in an initial video, determines a first optical flow from the first video frame to the second video frame and a second optical flow from the second video frame to the first video frame through a quantized neural network, and the first video frame and the second video frame are two adjacent initial video frames, so that the video frame adjustment method provided in the embodiments of the present disclosure can be run on a mobile device and is more robust for videos of large motion scenes. Based on the first optical flow and the second optical flow, a third optical flow from an intermediate frame to the first video frame and a fourth optical flow from the intermediate frame to the second video frame are determined, wherein the intermediate frame is an estimated video frame to be inserted between the first video frame and the second video frame; the intermediate frame is determined according to the first video frame, the second video frame, the third optical flow and the fourth optical flow; and the intermediate frame is inserted between the first video frame and the second video frame in the initial video. The purpose of inserting a frame between two adjacent video frames is achieved, and the playing effect of the high frame rate video after the frame insertion can be guaranteed. BRIEF DESCRIPTION OF DRAWINGS

[0024] The above and other features, advantages, and aspects of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals are used to refer to the same or similar elements. It is to be understood that the drawings are schematic, and the proportions of the elements and the sizes of the elements are not necessarily to scale.

[0025] Figure 1 A flowchart of a video frame adjustment method in the embodiments of the present disclosure;

[0026] Figure 2 A structural diagram of a quantized neural network in the embodiments of the present disclosure;

[0027] Figure 3 A timing relationship diagram between video frames in the embodiments of the present disclosure;

[0028] Figure 4 A diagram of a second optical flow vector of a second pixel point P from a first video frame I_0 to an intermediate frame I_t and a first optical flow vector of a first pixel point Q from the intermediate frame I_t to the first video frame I_0 in the embodiments of the present disclosure;

[0029] Figure 5 A flowchart of predicting an occlusion image through a preset neural network in the embodiments of the present disclosure

[0030] Figure 6 A structural diagram of a video frame adjustment device in the embodiments of the present disclosure; ​​

[0031] Figure 7 FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Embodiments of the present disclosure will be described in more detail with reference to the drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein, but rather should be construed in a broader sense. It is understood that the drawings and embodiments of the present disclosure are only for illustrative purposes and are not intended to limit the scope of the present disclosure.

[0033] It should be understood that each step described in the method embodiments of the present disclosure can be performed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0034] The term "comprising" and variations thereof as used in the present disclosure are open-ended, that is, "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Related definitions will be given in the description below.

[0035] It should be noted that the terms "first", "second", and the like mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0036] It should be noted that the modification of "one" or "multiple" in the present disclosure is illustrative and not limiting, and those skilled in the art should understand that unless otherwise explicitly stated in the context, it should be understood as "one or more".

[0037] The names of the messages or information exchanged between the devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0038] Figure 1 FIG. 2 is a flowchart of a video frame adjusting method according to an embodiment of the present disclosure. The method can be performed by a video frame adjusting device, which can be implemented in software and / or hardware, and can be configured in an electronic device, such as a terminal, including but not limited to a smartphone, a palm computer, a tablet computer, a wearable device with a display screen, a desktop computer, a notebook computer, an all-in-one computer, a smart home device with a display screen, etc.

[0039] As Figure 1 shown, the method can specifically include the following steps:

[0040] Step 110, based on the first video frame and the second video frame in the initial video, determining the first optical flow from the first video frame to the second video frame and the second optical flow from the second video frame to the first video frame by a quantized neural network, the first video frame and the second video frame being two adjacent initial video frames.

[0041] Wherein, the initial video refers to the original video obtained by shooting or recording by a shooting device, in other words, the initial video refers to the video which has not been processed by frame interpolation. The video frame in the initial video is the initial video frame. Two adjacent video frames refer to two video frames adjacent in time, for example, the first video frame is collected at 0 time, and the second video frame is collected at 1 time, then the first video frame and the second video frame are two video frames adjacent in time.

[0042] The quantized neural network can refer to a neural network that stores model parameters in INT8 (8-bit fixed-point integer). The meaning of quantization usually refers to converting the floating-point algorithm of the neural network into fixed-point, so that the neural network can run in real time on a mobile device (typically, for example, a smart phone), taking into account the performance limitations of the mobile device in terms of memory. By using a quantized neural network to estimate the motion of the target object in the two adjacent video frames, the video frame adjustment method provided in this embodiment can produce better frame interpolation effects for complex motion scenes, ensuring the playback effect of the target video obtained after frame interpolation; and by designing an efficient quantized neural network, the video frame adjustment method can run in real time on a mobile terminal, specifically, it can run in real time on a mobile terminal processor supporting quantized calculation, and real-time video frame adjustment is performed by the mobile terminal processor, achieving the purpose of improving the video frame rate, making the video play more smoothly.

[0043] In a specific embodiment, referring to as Figure 2An architecture diagram of a quantized neural network is shown, the quantized neural network includes a cascaded encoder module 210, a decoder module 220 and two optical flow prediction branches 230, and the two optical flow prediction branches 230 can be further divided into a first prediction branch 231 and a second prediction branch 232. The encoder module 210 includes a downsampling unit 211 and an encoding unit 212, the downsampling unit 211 is used to downsample two input video frames 200 (i.e. a first video frame and a second video frame) respectively, and input the downsampled images (i.e. a downsampled image of the first video frame and a downsampled image of the second video frame) corresponding to the two video frames 200 to the encoding unit 212, so that the encoding unit 212 extracts features based on the downsampled images, obtains the encoding of the feature images, and sends the encoding to the decoder module 220. The decoder module 220 includes a decoding unit and an upsampling unit, the decoding unit is used to decode the encoding of the feature images, and input the decoded feature images to the upsampling unit, so that the upsampling unit upsamples the decoded feature images, and inputs the obtained upsampled images to the two optical flow prediction branches 230 respectively, so that the two optical flow prediction branches 230 predict the first optical flow from the first video frame to the second video frame and the second optical flow from the second video frame to the first video frame based on the upsampled images respectively. For example, the first optical flow flow01 from the first video frame to the second video frame is predicted by the first prediction branch 231, and the second optical flow flow10 from the second video frame to the first video frame is predicted by the second prediction branch 232. That is, the input of the encoder module 210 is the first video frame I_0 and the second video frame I_1, before feature extraction by the encoding unit 212, the first video frame I_0 and the second video frame I_1 are downsampled by the downsampling unit 211 to improve the receptive field of the neural network. The advantage of this setting is that the neural network can be more robust for motion estimation in large motion scenes, and neural network inference on small resolution images can also improve inference speed. In order to make the neural network run more efficiently on mobile devices, the neural network is trained in a quantized manner, for example, an INT8 type quantized neural network can be finally obtained, which essentially uses INT8 type (8-bit fixed-point integer) to store model parameters.

[0044] Step 120, determining a third optical flow from the intermediate frame to the first video frame and a fourth optical flow from the intermediate frame to the second video frame based on the first optical flow and the second optical flow, wherein the intermediate frame is an estimated video frame to be inserted between the first video frame and the second video frame.

[0045] In an embodiment, determining, based on the first optical flow and the second optical flow, a third optical flow of the intermediate frame to the first video frame and a fourth optical flow of the intermediate frame to the second video frame comprises: determining, based on the first optical flow, the second optical flow and a motion trajectory of the target object between the first video frame and the second video frame, a fifth optical flow of the first video frame to the intermediate frame and a sixth optical flow of the second video frame to the intermediate frame; and determining, based on the fifth optical flow, the third optical flow and based on the sixth optical flow, the fourth optical flow by an optical flow inversion technique. Specifically, assuming that the first video frame is I_0 and the second video frame is I_1, and the intermediate frame is a video frame at a time t between the time 0 and the time 1, which is marked as I_t, a timing relationship between video frames can be referred to, for example, as shown in FIG. 10. Figure 3 The optical flow from the first video frame I_0 to the intermediate frame I_t is marked as the fifth optical flow, and the optical flow from the second video frame I_1 to the intermediate frame I_t is marked as the sixth optical flow. Taking the linear motion of the target object between the first video frame I_0 and the second video frame I_1 as an example, the fifth optical flow flow0t of the first video frame I_0 to the intermediate frame I_t is flow01*t, and the sixth optical flow flow1t of the second video frame I_1 to the intermediate frame I_t is flow10*(1-t). Wherein, flow01 represents the first optical flow from the first video frame I_0 to the second video frame I_1, and flow10 represents the second optical flow from the second video frame I_1 to the first video frame I_0, which can be obtained by the quantization neural network in step 110. After obtaining the fifth optical flow flow0t of the first video frame I_0 to the intermediate frame I_t and the sixth optical flow flow1t of the second video frame I_1 to the intermediate frame I_t, the third optical flow flowt0 of the intermediate frame I_t to the first video frame I_0 and the fourth optical flow flowt1 of the intermediate frame I_t to the second video frame I_1 can be determined by the optical flow inversion technique. The purpose of generating the third optical flow of the intermediate frame I_t to the first video frame I_0 and the fourth optical flow of the intermediate frame I_t to the second video frame I_1 is to ensure that each pixel on the intermediate frame I_t has a corresponding pixel on the first video frame I_0 and the second video frame I_1, so as to ensure the continuity of the video picture among the intermediate frame I_t, the first video frame I_0 and the second video frame I_1, thereby obtaining a better frame interpolation effect.

[0046] In an embodiment, the determining, based on the fifth optical flow and the sixth optical flow, the third optical flow by the optical flow inversion technique comprises:

[0047] If a first pixel in the intermediate frame has a predetermined relationship with a unique second pixel in the first video frame, then the first optical flow vector of the first pixel from the intermediate frame to the first video frame is the inverse vector of the second optical flow vector of the second pixel from the first video frame to the intermediate frame, where the fifth optical flow includes the second optical flow vector. The essence of the predetermined relationship between the first pixel in the intermediate frame and the unique second pixel in the first video frame is that the unique second pixel in the first video frame reaches the position of the first pixel in the intermediate frame when it moves from time 0 to time t. If a first pixel in the intermediate frame has a predetermined relationship with at least two second pixels in the first video frame, then the first optical flow vector is the weighted average of the inverse vectors of the second optical flow vectors of the at least two second pixels from the first video frame to the intermediate frame. The essence of the predetermined relationship between the first pixel in the intermediate frame and at least two second pixels in the first video frame is that at least two second pixels in the first video frame reach the position of the first pixel in the intermediate frame when they move from time 0 to time t. If there is no second pixel in the first video frame that has a preset relationship with the first pixel in the intermediate frame, then the first optical flow vector is 0. The essence of this is that none of the pixels in the first video frame have reached the position of the first pixel in the intermediate frame when they move from time 0 to time t. The first optical flow vector from the intermediate frame to the first video frame for each first pixel in the intermediate frame constitutes the third optical flow.

[0048] Specifically, for the second pixel P on the first video frame I_0, the second optical flow vector from the first video frame I_0 to the intermediate frame I_t is... The position of the second pixel P corresponding to the first pixel Q when it moves from the first video frame I_0 to the intermediate frame I_t can be calculated. Therefore, the optical flow vector of the first pixel Q from time t to time 0 is... That is, the first optical flow vector of the first pixel Q from the intermediate frame I_t to the first video frame I_0 is the second optical flow vector of the second pixel P from the first video frame I_0 to the intermediate frame I_t. The inverse vector is specifically: in, This represents the first optical flow vector. This represents the second optical flow vector. Correspondingly, see, for example... Figure 4 The diagram shows a second optical flow vector for a second pixel P from the first video frame I_0 to the intermediate frame I_t. And the first optical flow vector of the first pixel Q from the intermediate frame I_t to the first video frame I_0 A schematic diagram.

[0049] In one implementation, it is possible that the optical flow vectors of multiple second pixel points P on the first video frame I_0 arrive at the same first pixel point Q on the intermediate frame I_t. That is, the first pixel point Q corresponds to at least two second pixel points P, or, in other words, the first pixel point Q on the intermediate frame I_t has a preset relationship with at least two second pixel points P on the first video frame I_0. In this case, when calculating the first optical flow vector of the first pixel point Q from the intermediate frame I_t to the first video frame I_0, it is necessary to calculate the second optical flow vectors of the multiple second pixel points P arriving at the first pixel point Q from the first video frame I_0 to the intermediate frame I_t. The weighted average is calculated using the inverse vectors of the vectors, specifically as follows: Where N represents the second optical flow vector ending at the first pixel Q. The number of.

[0050] In one implementation, there may be a situation where no optical flow vector points to the first pixel Q, meaning there is no second pixel P corresponding to the first pixel Q. Alternatively, in other words, there may be no second pixel P on the first video frame I_0 that has a preset relationship with the first pixel Q on the intermediate frame I_t. In this case, the first optical flow vector is 0, and the first pixel Q is marked as an optical flow hole. The first optical flow vector from the intermediate frame to the first video frame for each first pixel Q on the intermediate frame constitutes the third optical flow.

[0051] The method for determining the fourth optical flow is similar to that for determining the third optical flow described above. Specifically, determining the fourth optical flow based on the sixth optical flow includes:

[0052] If the third pixel on the intermediate frame has a preset relationship with the unique fourth pixel on the second video frame, then the third optical flow vector of the third pixel from the intermediate frame to the second video frame is the inverse vector of the fourth optical flow vector of the fourth pixel from the second video frame to the intermediate frame, wherein the sixth optical flow includes the fourth optical flow vector.

[0053] If a third pixel on the intermediate frame has a preset relationship with at least two fourth pixels on the second video frame, then the third optical flow vector is the weighted average of the inverse vectors of the fourth optical flow vectors of the at least two fourth pixels from the second video frame to the intermediate frame. If there are no fourth pixels on the second video frame that have a preset relationship with the third pixel on the intermediate frame, then the third optical flow vector is 0. The third optical flow vector of each third pixel on the intermediate frame from the intermediate frame to the second video frame constitutes the fourth optical flow.

[0054] Step 130: Determine the intermediate frame based on the first video frame, the second video frame, the third optical flow, and the fourth optical flow.

[0055] In an implementation, the determining the intermediate frame according to the first video frame, the second video frame, the third optical flow and the fourth optical flow comprises:

[0056] performing image affine transformation on the first video frame to obtain a first transformed frame of the first video frame at a time of acquisition of the intermediate frame; performing image affine transformation on the second video frame to obtain a second transformed frame of the second video frame at the time of acquisition of the intermediate frame; predicting, by a preset neural network, a fusion weight of the first transformed frame and the second transformed frame based on a down-sampled image corresponding to the first video frame, a down-sampled image corresponding to the second video frame, the first transformed frame, the second transformed frame, the third optical flow and the fourth optical flow; and fusing, based on the fusion weight, pixels in the first transformed frame and the second transformed frame to obtain the intermediate frame, the fusion weight representing a probability that a pixel on the intermediate frame comes from the first video frame or from the second video frame.

[0057] wherein the first video frame I_0 is subjected to image affine transformation, i.e., warp transformation, to obtain a first transformed frame I_t of the first video frame at a time t of acquisition of the intermediate frame 0 The second video frame I_1 is subjected to image affine transformation, i.e., warp transformation, to obtain a second transformed frame I_t of the second video frame at the time t of acquisition of the intermediate frame 1 The purpose of the image affine transformation is to estimate a video frame of the first video frame I_0 at the time t and a video frame of the second video frame I_1 at the time t to provide a data source for obtaining the intermediate frame I_t.

[0058] To obtain the intermediate frame I_t, it is necessary to infer whether a certain pixel point in the intermediate frame comes from the first video frame I_0 or from the second video frame I_1. For this problem, in an implementation, a preset neural network is designed to predict a mask image mask (i.e., a fusion weight of the first transformed frame and the second transformed frame), a pixel value in the mask image mask ranges from 0 to 1, which represents a probability that the pixel point comes from the first video frame I_0, and the closer the pixel value is to 1, the greater the probability that the pixel point comes from the first video frame I_0. The input of the preset neural network includes the first transformed frame I_t 0 , the second transformed frame I_t 1, a third optical flow flowt0, a fourth optical flow flowt1, a down-sampled image corresponding to the first video frame I_0 and a down-sampled image corresponding to the second video frame I_1, and an output of the preset neural network is an occlusion image mask. In general, the preset neural network is used to predict a fusion weight of the first transformed frame and the second transformed frame based on the down-sampled image corresponding to the first video frame, the down-sampled image corresponding to the second video frame, the first transformed frame, the second transformed frame, the third optical flow and the fourth optical flow; and pixels in the first transformed frame and the second transformed frame are fused based on the fusion weight to obtain an intermediate frame, and the fusion weight represents a probability that a pixel on the intermediate frame comes from the first video frame or the second video frame. Correspondingly, reference can be made to a flowchart for predicting an occlusion image by using a preset neural network as shown in Figure 5 The down-sampled image corresponding to the first video frame, the down-sampled image corresponding to the second video frame, the first transformed frame, the second transformed frame, the third optical flow and the fourth optical flow are input into the mask network to obtain the occlusion image, i.e., the fusion weight of the first transformed frame and the second transformed frame.

[0059] Further, the pixels in the first transformed frame and the second transformed frame are fused based on the fusion weight (i.e., the pixel value in the occlusion image) to obtain the intermediate frame, including: obtaining the intermediate frame based on the following formula:

[0060]

[0061] wherein, I t represents the intermediate frame, mask represents the occlusion image, represents the first transformed frame, represents the second transformed frame, and the symbol "○" represents pixel-wise multiplication.

[0062] Step 140, inserting the intermediate frame between the first video frame and the second video frame in the initial video to obtain a target video.

[0063] The video frame adjustment method provided in the embodiment can estimate the motion of the target object in the two adjacent video frames by using the quantized neural network, so that the video frame adjustment method can produce better frame insertion effect for complex motion scenes and ensure the final playing effect of the video; and the efficient quantized neural network is designed, so that the video frame adjustment method can run in real time on a mobile terminal; the mask network is used to predict the occlusion image, so that the video frame adjustment method is more robust, and the fused intermediate frame is more natural and realistic.

[0064] It can be understood that not all adjacent two video frames are suitable for inserting intermediate frames, for example, reasonable intermediate frames cannot be estimated under lens switching or severe motion scenes, and therefore, in order to ensure the playing effect of the target video after the frame insertion, on the basis of the above-mentioned implementation solutions, in an implementation solution, before the intermediate frame is inserted between the first video frame and the second video frame, it further includes: determining whether the intermediate frame is suitable to be inserted between the first video frame and the second video frame based on the motion feature and / or color feature of the target object in the first video frame and the second video frame, if it is determined that the intermediate frame is suitable to be inserted between the first video frame and the second video frame, the step of inserting the intermediate frame between the first video frame and the second video frame is continued. If it is determined that the intermediate frame is not suitable to be inserted between the first video frame and the second video frame, the above-mentioned operation of inserting the intermediate frame between the first video frame and the second video frame is not performed, so as to avoid introducing artifacts in the obtained target video, thereby achieving the purpose of improving the video playing fluency while ensuring the video picture quality after the frame insertion. Specifically, whether the intermediate frame is suitable to be inserted between the first video frame and the second video frame can be determined by motion feature analysis, for example, by counting relevant indicators from color information and motion information.

[0065] Further, in an implementation solution, the motion feature of the target object in the first video frame and the second video frame includes at least one of the following: consistency of the third optical flow and the fourth optical flow; and number of optical flow void points in the intermediate frame, if there is no pixel point in the first video frame and the second video frame having a preset relationship with a specific pixel point in the intermediate frame, the specific pixel point is determined as an optical flow void point, that is, when there is no second pixel point P in the first video frame I_0 having a preset relationship with the first pixel point Q in the intermediate frame I_t, the first pixel point Q is marked as an optical flow void point.

[0066] The color feature of the target object in the first video frame and the second video frame includes a gray scale difference between the first transformed frame and the second transformed frame, wherein the first transformed frame is obtained by performing image affine transformation on the first video frame, and the second transformed frame is obtained by performing image affine transformation on the second video frame.

[0067] Specifically, if the motion feature of the target object in the first video frame and the second video frame is the consistency of the third optical flow and the fourth optical flow, whether the intermediate frame is suitable to be inserted between the first video frame and the second video frame is determined based on the consistency of the third optical flow and the fourth optical flow, including:

[0068] For linear motion, a linear motion offset distance is determined according to a forward motion vector of a target pixel point on the intermediate frame moving from the intermediate frame to the first video frame and a backward motion vector of the target pixel point moving from the intermediate frame to the second video frame; a pixel number ratio that is greater than a first set threshold is counted; if the pixel number ratio is less than or equal to a second set threshold, it is determined that the intermediate frame is suitable to be inserted between the first video frame and the second video frame; if the pixel number ratio is greater than the second set threshold, it is determined that the intermediate frame is not suitable to be inserted between the first video frame and the second video frame. Whether the optical flow is reliable is determined by calculating the consistency of the third optical flow flowt0 of the intermediate frame I_t to the first video frame I_0 and the fourth optical flow flowt1 of the intermediate frame I_t to the second video frame I_1. Under the assumption of linear motion, the forward motion vector of a certain pixel point (i.e. the target pixel point) on the intermediate frame moving from the intermediate frame to the first video frame and the backward motion vector of the target pixel point moving from the intermediate frame to the second video frame should be equal in size and opposite in direction. For a certain pixel point Q on the intermediate frame, the forward motion vector f t0 and the backward motion vector f t1 of the target pixel point can be calculated according to the following formula: t0 distance = ‖f t1 ‖ 2 The linear motion offset distance is calculated: distance = ‖f t0 +f t1 ‖ 2 The proportion of the pixel points whose linear motion offset distance is greater than the first set threshold, i.e. the proportion of the number of the pixel points whose linear motion offset distance is greater than the first set threshold in the total number of the pixel points on the intermediate frame, is counted. If the proportion is greater than a second set threshold, it is determined that the intermediate frame is not suitable to be inserted between the first video frame and the second video frame.

[0069] In an embodiment, if the gray difference between the first transformed frame and the second transformed frame is greater than a third set threshold, it is determined that the intermediate frame is not suitable to be inserted between the first video frame and the second video frame.

[0070] In an embodiment, the optical flow empty points in the intermediate frame have been marked in an embodiment of the above step 120. These optical flow empty points often occur in the occlusion area. The number of the optical flow empty points is counted. The greater the number, the greater the area of the occlusion area. If the area of the occlusion area is too large, the frame insertion is prone to errors. Therefore, in order to ensure the quality of the video picture, the frame insertion is not performed between the first video frame and the second video frame in this case, so as to avoid introducing artifacts in the target video obtained after the frame insertion, while improving the smoothness of the video playing and ensuring the quality of the video picture after the frame insertion.

[0071] Figure 6 A structural schematic diagram of a video frame adjustment device in an embodiment of the present disclosure is shown. The video frame adjustment device provided by the embodiment of the present disclosure can be configured in a terminal. As shown in the figure, the video frame adjustment device comprises a video frame adjustment unit 100, a video frame adjustment unit 200, a video frame adjustment unit 300, a video frame adjustment unit 400 and a video frame adjustment unit 500. Figure 6As shown, the video frame adjusting apparatus specifically comprises a first determining module 610, a second determining module 620, a third determining module 630 and an interpolation module 640.

[0072] The first determining module 610 is configured to determine, by a quantized neural network, a first optical flow from the first video frame to the second video frame and a second optical flow from the second video frame to the first video frame based on the first video frame and the second video frame in the initial video, the first video frame and the second video frame being two adjacent initial video frames. The second determining module 620 is configured to determine a third optical flow from an intermediate frame to the first video frame and a fourth optical flow from the intermediate frame to the second video frame based on the first optical flow and the second optical flow, the intermediate frame being an estimated video frame to be inserted between the first video frame and the second video frame. The third determining module 630 is configured to determine the intermediate frame according to the first video frame, the second video frame, the third optical flow and the fourth optical flow. The interpolation module 640 is configured to insert the intermediate frame into the initial video between the first video frame and the second video frame to obtain a target video.

[0073] Optionally, the second determining module 620 comprises:

[0074] The first determining unit is configured to determine a fifth optical flow from the first video frame to the intermediate frame and a sixth optical flow from the second video frame to the intermediate frame based on the first optical flow, the second optical flow and a motion trajectory of a target object in the first video frame and the second video frame. The second determining unit is configured to determine the third optical flow based on the fifth optical flow and determine the fourth optical flow based on the sixth optical flow by an optical flow reversal technology.

[0075] Optionally, the second determining unit comprises:

[0076] The first determining sub-unit is configured to: if the first pixel point on the intermediate frame has a preset relationship with a unique second pixel point on the first video frame, a first optical flow vector of the first pixel point from the intermediate frame to the first video frame is an inverse vector of a second optical flow vector of the second pixel point from the first video frame to the intermediate frame, wherein the fifth optical flow comprises the second optical flow vector; if the first pixel point on the intermediate frame has a preset relationship with at least two second pixel points on the first video frame, the first optical flow vector is a weighted average value of inverse vectors of the second optical flow vectors of the at least two second pixel points from the first video frame to the intermediate frame; if there is no second pixel point on the first video frame having a preset relationship with the first pixel point on the intermediate frame, the first optical flow vector is 0; and first optical flow vectors of each of the first pixel points on the intermediate frame from the intermediate frame to the first video frame form the third optical flow.

[0077] Optionally, the second determining unit further comprises:

[0078] The second determining sub-unit is configured to: if the third pixel point on the intermediate frame has a preset relationship with a unique fourth pixel point on the second video frame, a third optical flow vector of the third pixel point from the intermediate frame to the second video frame is an inverse vector of a fourth optical flow vector of the fourth pixel point from the second video frame to the intermediate frame, wherein the sixth optical flow comprises the fourth optical flow vector; if the third pixel point on the intermediate frame has a preset relationship with at least two fourth pixel points on the second video frame, the third optical flow vector is a weighted average value of inverse vectors of the fourth optical flow vectors of the at least two fourth pixel points from the second video frame to the intermediate frame; if there is no fourth pixel point on the second video frame having a preset relationship with the third pixel point on the intermediate frame, the third optical flow vector is 0; and third optical flow vectors of each of the third pixel points on the intermediate frame from the intermediate frame to the second video frame form the fourth optical flow.

[0079] Optionally, the quantization neural network comprises a cascaded encoder module, a decoder module and two optical flow prediction branches; wherein the encoder module comprises a down-sampling unit and an encoding unit, the down-sampling unit is configured to down-sample the input first video frame and the input second video frame respectively, and input the down-sampled image of the first video frame and the down-sampled image of the second video frame to the encoding unit, so that the encoding unit extracts features based on the down-sampled images to obtain the encoding of the feature image; the decoder module comprises a decoding unit and an up-sampling unit, the decoding unit is configured to decode the encoding of the feature image, and input the decoded feature image to the up-sampling unit, so that the up-sampling unit up-samples the decoded feature image, and inputs the obtained up-sampled image to the two optical flow prediction branches, so that one of the two optical flow prediction branches predicts the first optical flow based on the up-sampled image, and the other of the two optical flow prediction branches predicts the second optical flow based on the up-sampled image.

[0080] Optionally, the third determination module 630 comprises:

[0081] a transformation unit configured to perform image affine transformation on the first video frame to obtain a first transformed frame of the first video frame at the acquisition time of the intermediate frame, and perform image affine transformation on the second video frame to obtain a second transformed frame of the second video frame at the acquisition time of the intermediate frame; a prediction unit configured to predict a fusion weight of the first transformed frame and the second transformed frame based on the down-sampled image corresponding to the first video frame, the down-sampled image corresponding to the second video frame, the first transformed frame, the second transformed frame, the third optical flow and the fourth optical flow through a preset neural network; and a fusion unit configured to fuse pixels in the first transformed frame and the second transformed frame based on the fusion weight to obtain the intermediate frame, the fusion weight representing a probability that a pixel on the intermediate frame comes from the first video frame or from the second video frame.

[0082] Optionally, the video frame adjusting apparatus further comprises:

[0083] The determination module is configured to determine whether the intermediate frame is suitable to be inserted between the first video frame and the second video frame based on motion features and / or color features of the target object in the first video frame and the second video frame before the intermediate frame is inserted between the first video frame and the second video frame, and if it is determined that the intermediate frame is suitable to be inserted between the first video frame and the second video frame, the step of inserting the intermediate frame between the first video frame and the second video frame is continued. Optionally, the motion features of the target object in the first video frame and the second video frame include at least one of the following: consistency of the third optical flow and the fourth optical flow; and a number of optical flow void points in the intermediate frame, wherein if there is no pixel point in the first video frame and the second video frame that has a preset relationship with a specific pixel point in the intermediate frame, the specific pixel point is determined as an optical flow void point. The color features of the target object in the first video frame and the second video frame include a gray scale difference between the first transformed frame and the second transformed frame, wherein the first transformed frame is obtained by performing image affine transformation on the first video frame, and the second transformed frame is obtained by performing image affine transformation on the second video frame.

[0084] Optionally, if the motion features of the target object in the first video frame and the second video frame are consistency of the third optical flow and the fourth optical flow, the determination module is specifically configured to: for linear motion, determine a linear motion offset distance according to a forward motion vector of a target pixel point in the intermediate frame moving from the intermediate frame to the first video frame and a backward motion vector of the target pixel point moving from the intermediate frame to the second video frame; and count a pixel number ratio of the linear motion offset distance greater than a first set threshold value; if the pixel number ratio is less than or equal to a second set threshold value, it is determined that the first corresponding video image is suitable.

[0085] The video frame adjustment apparatus provided by the embodiments of the present disclosure can estimate the motion of the target object in the two adjacent video frames by using the quantized neural network, so that the video frame adjustment method can produce better frame insertion effect for complex motion scenes and ensure the final playing effect of the video. In addition, the efficient quantized neural network is designed, so that the video frame adjustment method can run in real time on a mobile terminal. The mask network is used to predict the occlusion image, so that the video frame adjustment method is more robust, and the fused intermediate frame is more natural and realistic. By adding the adaptive frame insertion determination algorithm, it is determined whether the intermediate frame is suitable to be inserted between the two adjacent video frames before the frame insertion. If not, the intermediate frame is not inserted between the two video frames, so as to avoid introducing motion artifacts and achieve the purpose of improving the video frame rate while ensuring the video picture quality.

[0086] The video frame adjustment apparatus provided in this disclosure can execute the steps in the video frame adjustment method provided in this disclosure, and has the execution steps and beneficial effects, which will not be described in detail here.

[0087] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this disclosure. See below for details. Figure 7 The diagram illustrates a structural schematic suitable for implementing the electronic device 500 in the embodiments of this disclosure. The electronic device 500 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0088] like Figure 7 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 501, which can perform various appropriate actions and processes to implement the methods as described in the embodiments of this disclosure, based on a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing device 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0089] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0090] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts, thereby implementing the video frame adjustment method as described above. In such embodiments, the computer program can be downloaded and installed from a network by the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0091] It should be noted that the computer-readable medium described above in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used or used in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take on many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to a wire, cable, optical fiber, RF (radio frequency), or the like, or any suitable combination thereof.

[0092] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed network.

[0093] The computer-readable medium described above can be included in the electronic device described above; alternatively, it can exist separately from the electronic device and be not assembled into the electronic device.

[0094] The computer-readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to:

[0095] Based on a first video frame and a second video frame in an initial video, determining, by a quantized neural network, a first optical flow from the first video frame to the second video frame and a second optical flow from the second video frame to the first video frame, the first video frame and the second video frame being two adjacent initial video frames; determining, based on the first optical flow and the second optical flow, a third optical flow from an intermediate frame to the first video frame and a fourth optical flow from the intermediate frame to the second video frame, the intermediate frame being an estimated video frame to be inserted between the first video frame and the second video frame; determining the intermediate frame according to the first video frame, the second video frame, the third optical flow, and the fourth optical flow; and inserting the intermediate frame between the first video frame and the second video frame in the initial video to obtain a target video.

[0096] Optionally, when the one or more programs described above are executed by the electronic device, the electronic device can further perform other steps described in the above embodiments.

[0097] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0098] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0099] The units described in the embodiments of the present disclosure can be implemented by hardware, software, or a combination of hardware and software. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0100] The functions described in this specification can be implemented in part or in whole through one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0101] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0102] According to one or more embodiments of the present disclosure, the present disclosure provides a video frame adjustment method, which comprises: determining, by a quantization neural network, a first optical flow from a first video frame to a second video frame and a second optical flow from the second video frame to the first video frame based on the first video frame and the second video frame in an initial video, the first video frame and the second video frame being two adjacent initial video frames; determining a third optical flow from an intermediate frame to the first video frame and a fourth optical flow from the intermediate frame to the second video frame based on the first optical flow and the second optical flow, wherein the intermediate frame is an estimated video frame to be inserted between the first video frame and the second video frame; determining the intermediate frame according to the first video frame, the second video frame, the third optical flow and the fourth optical flow; and inserting the intermediate frame between the first video frame and the second video frame in the initial video to obtain a target video.

[0103] According to one or more embodiments of the present disclosure, in the video frame adjustment method provided by the present disclosure, optionally, the determining the third optical flow from the intermediate frame to the first video frame and the fourth optical flow from the intermediate frame to the second video frame based on the first optical flow and the second optical flow comprises: determining a fifth optical flow from the first video frame to the intermediate frame and a sixth optical flow from the second video frame to the intermediate frame based on the first optical flow, the second optical flow and a motion trajectory of a target object in the first video frame and the second video frame; and determining the third optical flow based on the fifth optical flow and determining the fourth optical flow based on the sixth optical flow by an optical flow reversal technology.

[0104] According to one or more embodiments of the present disclosure, in the video frame adjustment method provided by the present disclosure, optionally, the third optical flow is determined based on the fifth optical flow by using the optical flow inversion technology, including: if the first pixel point on the intermediate frame has a preset relationship with a unique second pixel point on the first video frame, then the first optical flow vector of the first pixel point from the intermediate frame to the first video frame is the inverse vector of the second optical flow vector of the second pixel point from the first video frame to the intermediate frame, wherein the fifth optical flow includes the second optical flow vector; if the first pixel point on the intermediate frame has a preset relationship with at least two second pixel points on the first video frame, then the first optical flow vector is the weighted average value of the inverse vectors of the second optical flow vectors of the at least two second pixel points from the first video frame to the intermediate frame; if there is no second pixel point on the first video frame that has a preset relationship with the first pixel point on the intermediate frame, then the first optical flow vector is 0; and the first optical flow vectors of each of the first pixel points on the intermediate frame from the intermediate frame to the first video frame constitute the third optical flow.

[0105] According to one or more embodiments of the present disclosure, in the video frame adjustment method provided by the present disclosure, optionally, the fourth optical flow is determined based on the sixth optical flow, including: if the third pixel point on the intermediate frame has a preset relationship with a unique fourth pixel point on the second video frame, then the third optical flow vector of the third pixel point from the intermediate frame to the second video frame is the inverse vector of the fourth optical flow vector of the fourth pixel point from the second video frame to the intermediate frame, wherein the sixth optical flow includes the fourth optical flow vector; if the third pixel point on the intermediate frame has a preset relationship with at least two fourth pixel points on the second video frame, then the third optical flow vector is the weighted average value of the inverse vectors of the fourth optical flow vectors of the at least two fourth pixel points from the second video frame to the intermediate frame; if there is no fourth pixel point on the second video frame that has a preset relationship with the third pixel point on the intermediate frame, then the third optical flow vector is 0; and the third optical flow vectors of each of the third pixel points on the intermediate frame from the intermediate frame to the second video frame constitute the fourth optical flow.

[0106] According to one or more embodiments of the present disclosure, in the video frame adjustment method provided by the present disclosure, optionally, the quantization neural network comprises a cascaded encoder module, a decoder module and two optical flow prediction branches; wherein the encoder module comprises a downsampling unit and an encoding unit, the downsampling unit is configured to downsample the input first video frame and the input second video frame respectively, and input the downsampled image of the first video frame and the downsampled image of the second video frame to the encoding unit, so that the encoding unit extracts features based on the downsampled images to obtain the encoding of the feature image; the decoder module comprises a decoding unit and an upsampling unit, the decoding unit is configured to decode the encoding of the feature image, and input the decoded feature image to the upsampling unit, so that the upsampling unit upsamples the decoded feature image, and inputs the obtained upsampled image to the two optical flow prediction branches, so that one of the two optical flow prediction branches predicts the first optical flow based on the upsampled image, and the other of the two optical flow prediction branches predicts the second optical flow based on the upsampled image.

[0107] According to one or more embodiments of the present disclosure, in the video frame adjustment method provided by the present disclosure, optionally, the determining the intermediate frame according to the first video frame, the second video frame, the third optical flow and the fourth optical flow comprises: performing image affine transformation on the first video frame to obtain a first transformed frame of the first video frame at the acquisition time of the intermediate frame; performing image affine transformation on the second video frame to obtain a second transformed frame of the second video frame at the acquisition time of the intermediate frame; predicting fusion weights of the first transformed frame and the second transformed frame based on the downsampled image corresponding to the first video frame, the downsampled image corresponding to the second video frame, the first transformed frame, the second transformed frame, the third optical flow and the fourth optical flow through a preset neural network; and fusing pixels in the first transformed frame and the second transformed frame based on the fusion weights to obtain the intermediate frame, wherein the fusion weights represent the probability that a pixel on the intermediate frame comes from the first video frame or from the second video frame.

[0108] According to one or more embodiments of the present disclosure, in the video frame adjustment method provided by the present disclosure, optionally, before the inserting the intermediate frame between the first video frame and the second video frame, the method further comprises: determining whether it is appropriate to insert the intermediate frame between the first video frame and the second video frame based on the motion characteristics and / or color characteristics of the target object in the first video frame and the second video frame, and if it is determined that it is appropriate to insert the intermediate frame between the first video frame and the second video frame, then the step of inserting the intermediate frame between the first video frame and the second video frame is continued.

[0109] According to one or more embodiments of the present disclosure, in the video frame adjustment method provided by the present disclosure, the motion feature of the target object in the first video frame and the second video frame includes at least one of the following: consistency of the third optical flow and the fourth optical flow; number of optical flow void points in the intermediate frame, if there is no pixel point in the first video frame and the second video frame having a preset relationship with a specific pixel point in the intermediate frame, the specific pixel point is determined as an optical flow void point; the color feature of the target object in the first video frame and the second video frame includes a gray scale difference between the first transformed frame and the second transformed frame, wherein the first transformed frame is obtained by performing image affine transformation on the first video frame, and the second transformed frame is obtained by performing image radiation transformation on the second video frame.

[0110] According to one or more embodiments of the present disclosure, in the video frame adjustment method provided by the present disclosure, if the motion feature of the target object in the first video frame and the second video frame is the consistency of the third optical flow and the fourth optical flow, whether the intermediate frame is suitable to be inserted between the first video frame and the second video frame is determined based on the consistency of the third optical flow and the fourth optical flow, including: for linear motion, determining a linear motion offset distance according to a forward motion vector of a target pixel point in the intermediate frame moving from the intermediate frame to the first video frame, and a backward motion vector of the target pixel point moving from the intermediate frame to the second video frame; statistics the proportion of the number of pixels whose linear motion offset distance is greater than a first set threshold; if the proportion of the number of pixels is less than or equal to a second set threshold, it is determined that the first corresponding video image is suitable.

[0111] According to one or more embodiments of the present disclosure, the present disclosure provides a video frame adjustment device, which comprises: a first determination module, configured to determine a first optical flow from a first video frame to a second video frame and a second optical flow from the second video frame to the first video frame based on the first video frame and the second video frame in an initial video, the first video frame and the second video frame being two adjacent initial video frames; a second determination module, configured to determine a third optical flow from an intermediate frame to the first video frame and a fourth optical flow from the intermediate frame to the second video frame based on the first optical flow and the second optical flow, wherein the intermediate frame is an estimated video frame to be inserted between the two initial video frames; a third determination module, configured to determine the intermediate frame according to the first video frame, the second video frame, the third optical flow and the fourth optical flow; an interpolation module, configured to insert the video image between the two initial video frames in the initial video to obtain a target video.

[0112] According to one or more embodiments of the present disclosure, in the video frame adjusting device provided by the present disclosure, the second determining module comprises: a first determining unit configured to determine a fifth optical flow from the first video frame to the intermediate frame and a sixth optical flow from the second video frame to the intermediate frame based on the first optical flow, the second optical flow and the motion trajectory of the target object in the first video frame and the second video frame; and a second determining unit configured to determine the third optical flow based on the fifth optical flow and determine the fourth optical flow based on the sixth optical flow by using an optical flow reversal technology.

[0113] According to one or more embodiments of the present disclosure, in the video frame adjusting device provided by the present disclosure, the second determining unit comprises: a first determining sub-unit configured to, if a first pixel point on the intermediate frame has a preset relationship with a unique second pixel point on the first video frame, determine a first optical flow vector of the first pixel point from the intermediate frame to the first video frame as an inverse vector of a second optical flow vector of the second pixel point from the first video frame to the intermediate frame, wherein the fifth optical flow comprises the second optical flow vector; if a first pixel point on the intermediate frame has a preset relationship with at least two second pixel points on the first video frame, determine the first optical flow vector as a weighted average value of inverse vectors of second optical flow vectors of the at least two second pixel points from the first video frame to the intermediate frame; if there is no second pixel point on the first video frame having a preset relationship with a first pixel point on the intermediate frame, determine the first optical flow vector as 0; and determine the third optical flow as a set of first optical flow vectors of each first pixel point on the intermediate frame from the intermediate frame to the first video frame.

[0114] According to one or more embodiments of the present disclosure, in the video frame adjusting device provided by the present disclosure, the second determining unit further comprises: a second determining sub-unit configured to, if a third pixel point on the intermediate frame has a preset relationship with a unique fourth pixel point on the second video frame, determine a third optical flow vector of the third pixel point from the intermediate frame to the second video frame as an inverse vector of a fourth optical flow vector of the fourth pixel point from the second video frame to the intermediate frame, wherein the sixth optical flow comprises the fourth optical flow vector; if a third pixel point on the intermediate frame has a preset relationship with at least two fourth pixel points on the second video frame, determine the third optical flow vector as a weighted average value of inverse vectors of fourth optical flow vectors of the at least two fourth pixel points from the second video frame to the intermediate frame; if there is no fourth pixel point on the second video frame having a preset relationship with a third pixel point on the intermediate frame, determine the third optical flow vector as 0; and determine the fourth optical flow as a set of third optical flow vectors of each third pixel point on the intermediate frame from the intermediate frame to the second video frame.

[0115] According to one or more embodiments of the present disclosure, in the video frame adjusting apparatus provided by the present disclosure, the optional quantization neural network comprises a cascaded encoder module, a decoder module and two optical flow prediction branches; the encoder module comprises a downsampling unit and an encoding unit, the downsampling unit is configured to downsample the input first video frame and the input second video frame respectively, and input the downsampled image of the first video frame and the downsampled image of the second video frame to the encoding unit, so that the encoding unit extracts features based on the downsampled images to obtain the encoding of the feature image; the decoder module comprises a decoding unit and an upsampling unit, the decoding unit is configured to decode the encoding of the feature image and input the decoded feature image to the upsampling unit, so that the upsampling unit upsamples the decoded feature image and inputs the obtained upsampled image to the two optical flow prediction branches, so that one of the two optical flow prediction branches predicts the first optical flow based on the upsampled image, and the other of the two optical flow prediction branches predicts the second optical flow based on the upsampled image.

[0116] According to one or more embodiments of the present disclosure, in the video frame adjusting apparatus provided by the present disclosure, the optional third determination module comprises: a transformation unit configured to perform image affine transformation on the first video frame to obtain a first transformed frame of the first video frame at the acquisition time of the intermediate frame, and perform image affine transformation on the second video frame to obtain a second transformed frame of the second video frame at the acquisition time of the intermediate frame; a prediction unit configured to predict a fusion weight of the first transformed frame and the second transformed frame based on the downsampled image corresponding to the first video frame, the downsampled image corresponding to the second video frame, the first transformed frame, the second transformed frame, the third optical flow and the fourth optical flow through a preset neural network; and a fusion unit configured to fuse pixels in the first transformed frame and the second transformed frame based on the fusion weight to obtain the intermediate frame, the fusion weight representing the probability that a pixel on the intermediate frame comes from the first video frame or from the second video frame.

[0117] According to one or more embodiments of the present disclosure, in the video frame adjusting apparatus provided by the present disclosure, the optional further comprises: a determination module configured to determine whether it is appropriate to insert the intermediate frame between the first video frame and the second video frame based on the motion characteristics and / or color characteristics of the target object in the first video frame and the second video frame before the step of inserting the intermediate frame between the first video frame and the second video frame, and if it is determined that it is appropriate to insert the intermediate frame between the first video frame and the second video frame, the step of inserting the intermediate frame between the first video frame and the second video frame is continued.

[0118] According to one or more embodiments of the present disclosure, in the video frame adjustment apparatus provided by the present disclosure, the motion feature of the target object in the first video frame and the second video frame includes at least one of the following: consistency of the third optical flow and the fourth optical flow; and a number of optical flow void points in the intermediate frame, wherein if there is no pixel point in the first video frame and the second video frame having a preset relationship with a specific pixel point in the intermediate frame, the specific pixel point is determined as an optical flow void point; and the color feature of the target object in the first video frame and the second video frame includes a gray scale difference between the first transformed frame and the second transformed frame, wherein the first transformed frame is obtained by performing image affine transformation on the first video frame, and the second transformed frame is obtained by performing image radiation transformation on the second video frame.

[0119] According to one or more embodiments of the present disclosure, in the video frame adjustment apparatus provided by the present disclosure, if the motion feature of the target object in the first video frame and the second video frame is the consistency of the third optical flow and the fourth optical flow, the determination module is specifically configured to: for linear motion, determine a linear motion offset distance according to a forward motion vector of a target pixel point on the intermediate frame moving from the intermediate frame to the first video frame, and a backward motion vector of the target pixel point moving from the intermediate frame to the second video frame; and count a pixel number ratio that is greater than a first set threshold; and if the pixel number ratio is less than or equal to a second set threshold, determine the first corresponding video image.

[0120] According to one or more embodiments of the present disclosure, an electronic device is provided, which includes: one or more processors; a memory configured to store one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the video frame adjustment method provided by any one of the embodiments of the present disclosure.

[0121] According to one or more embodiments of the present disclosure, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement any one of the video frame adjustment methods provided by the embodiments of the present disclosure.

[0122] The embodiments of the present disclosure further provide a computer program product, which includes a computer program or instructions, and the computer program or instructions are executed by a processor to implement the video frame adjustment method as described above.

[0123] The above description merely illustrates the preferred embodiments of the disclosure and a principle for applying the technologies. It is understood by those skilled in the art that the disclosed scope of the disclosure is not limited to the technical solutions formed by the specific combinations of the technical features described above, and should also cover other technical solutions formed by the combinations of the technical features described above or their equivalent features without departing from the disclosed concept. For example, the technical solutions formed by the mutual replacement of the above-described features and the technical features with similar functions disclosed in the disclosure (but not limited to) can be formed.

[0124] Further, although operations are depicted in a particular, sequential order, this should not be understood as requiring or implying that the operations are performed in the order illustrated or sequentially. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, although specific implementation details are included for the purpose of providing a thorough disclosure, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0125] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method of video frame adjustment, the method comprising: The method comprises: determining, based on a first video frame and a second video frame in an initial video, a first optical flow from the first video frame to the second video frame and a second optical flow from the second video frame to the first video frame by a quantization neural network, the first video frame and the second video frame being two adjacent initial video frames; determining a third optical flow from an intermediate frame to the first video frame and a fourth optical flow from the intermediate frame to the second video frame based on the first optical flow and the second optical flow, wherein the intermediate frame is an estimated video frame to be inserted between the first video frame and the second video frame; determining the intermediate frame according to the first video frame, the second video frame, the third optical flow and the fourth optical flow; inserting the intermediate frame between the first video frame and the second video frame in the initial video to obtain a target video; the determining the intermediate frame according to the first video frame, the second video frame, the third optical flow and the fourth optical flow comprises: performing image affine transformation on the first video frame to obtain a first transformed frame of the first video frame at a time when the intermediate frame is acquired; performing image affine transformation on the second video frame to obtain a second transformed frame of the second video frame at the time when the intermediate frame is acquired; predicting a fusion weight of the first transformed frame and the second transformed frame based on a down-sampled image corresponding to the first video frame, a down-sampled image corresponding to the second video frame, the first transformed frame, the second transformed frame, the third optical flow and the fourth optical flow by a preset neural network; fusing pixels in the first transformed frame and the second transformed frame based on the fusion weight to obtain the intermediate frame, the fusion weight representing a probability that a pixel on the intermediate frame comes from the first video frame or from the second video frame.

2. The method of claim 1, wherein, the determining the third optical flow from the intermediate frame to the first video frame and the fourth optical flow from the intermediate frame to the second video frame based on the first optical flow and the second optical flow comprises: determining a fifth optical flow from the first video frame to the intermediate frame and a sixth optical flow from the second video frame to the intermediate frame based on the first optical flow, the second optical flow and a motion trajectory of a target object in the first video frame and the second video frame; determining the third optical flow based on the fifth optical flow and determining the fourth optical flow based on the sixth optical flow by an optical flow inversion technology.

3. The method of claim 2, wherein, the determining the third optical flow based on the fifth optical flow by the optical flow inversion technology comprises: if a first pixel point on the intermediate frame has a preset relationship with a unique second pixel point on the first video frame, a first optical flow vector of the first pixel point from the intermediate frame to the first video frame is an inverse vector of a second optical flow vector of the second pixel point from the first video frame to the intermediate frame, wherein the fifth optical flow comprises the second optical flow vector. if the first pixel point on the intermediate frame has a preset relationship with at least two second pixel points on the first video frame, the first optical flow vector is a weighted average of inverse vectors of second optical flow vectors of the at least two second pixel points from the first video frame to the intermediate frame respectively; if there is no second pixel point on the first video frame having a preset relationship with the first pixel point on the intermediate frame, the first optical flow vector is 0; a first optical flow vector group of each of the first pixel points on the intermediate frame from the intermediate frame to the first video frame constitutes the third optical flow.

4. The method of claim 2, wherein, The determining the fourth optical flow based on the sixth optical flow comprises: if a third pixel point on the intermediate frame has a preset relationship with a unique fourth pixel point on the second video frame, a third optical flow vector of the third pixel point from the intermediate frame to the second video frame is an inverse vector of a fourth optical flow vector of the fourth pixel point from the second video frame to the intermediate frame, wherein the sixth optical flow comprises the fourth optical flow vector; if a third pixel point on the intermediate frame has a preset relationship with at least two fourth pixel points on the second video frame, the third optical flow vector is a weighted average of inverse vectors of fourth optical flow vectors of the at least two fourth pixel points from the second video frame to the intermediate frame respectively; if there is no fourth pixel point on the second video frame having a preset relationship with the third pixel point on the intermediate frame, the third optical flow vector is 0; a third optical flow vector of each of the third pixel points on the intermediate frame from the intermediate frame to the second video frame constitutes the fourth optical flow.

5. The method of claim 1, wherein, The quantization neural network comprises a cascaded encoder module, a decoder module and two optical flow prediction branches; The encoder module comprises a downsampling unit and an encoding unit, the downsampling unit is configured to downsample the input first video frame and the input second video frame respectively, and input the downsampled image of the first video frame and the downsampled image of the second video frame to the encoding unit, so that the encoding unit extracts features based on the downsampled images to obtain an encoding of a feature image; The decoder module comprises a decoding unit and an upsampling unit, the decoding unit is configured to decode the encoding of the feature image, and input the decoded feature image to the upsampling unit, so that the upsampling unit up-samples the decoded feature image, and input the obtained up-sampled image to the two optical flow prediction branches respectively, so that one of the two optical flow prediction branches predicts the first optical flow based on the up-sampled image, and the other of the two optical flow prediction branches predicts the second optical flow based on the up-sampled image.

6. The method according to any one of claims 1 to 5, characterized in that, Before the inserting the intermediate frame between the first video frame and the second video frame, the method further comprises: determine whether the intermediate frame is suitable to be inserted between the first video frame and the second video frame based on motion features and / or color features of the target object in the first video frame and the second video frame, and if it is determined that the intermediate frame is suitable to be inserted between the first video frame and the second video frame, continue to perform the step of inserting the intermediate frame between the first video frame and the second video frame.

7. The method of claim 6, wherein, The motion features of the target object in the first video frame and the second video frame include at least one of the following: consistency of the third optical flow and the fourth optical flow; a number of optical flow void points in the intermediate frame, wherein if there is no pixel point in the first video frame and the second video frame that has a preset relationship with a specific pixel point in the intermediate frame, the specific pixel point is determined as an optical flow void point; The color features of the target object in the first video frame and the second video frame include: a gray scale difference between the first transformed frame and the second transformed frame, wherein the first transformed frame is obtained by performing image affine transformation on the first video frame, and the second transformed frame is obtained by performing image affine transformation on the second video frame.

8. The method of claim 6, wherein, If the motion features of the target object in the first video frame and the second video frame are the consistency of the third optical flow and the fourth optical flow, the determination of whether the intermediate frame is suitable to be inserted between the first video frame and the second video frame based on the consistency of the third optical flow and the fourth optical flow includes: for linear motion, determining a linear motion offset distance according to a forward motion vector of a target pixel point on the intermediate frame moving from the intermediate frame to the first video frame, and a backward motion vector of the target pixel point moving from the intermediate frame to the second video frame; statistically determining a pixel number ratio of the linear motion offset distance greater than a first set threshold value; if the pixel number ratio is less than or equal to a second set threshold value, it is determined that the intermediate frame is suitable to be inserted between the first video frame and the second video frame.

9. A video frame adjustment apparatus, characterized by comprising: including: a first determination module configured to determine, based on a first video frame and a second video frame in an initial video, a first optical flow from the first video frame to the second video frame and a second optical flow from the second video frame to the first video frame by a quantization neural network, the first video frame and the second video frame being two adjacent initial video frames; a second determination module configured to determine, based on the first optical flow and the second optical flow, a third optical flow from an intermediate frame to the first video frame and a fourth optical flow from the intermediate frame to the second video frame, wherein the intermediate frame is an estimated video frame to be inserted between the first video frame and the second video frame; a third determination module configured to determine the intermediate frame according to the first video frame, the second video frame, the third optical flow and the fourth optical flow; an interpolation module configured to insert the intermediate frame between the first video frame and the second video frame in the initial video to obtain a target video; the third determination module includes: a transformation unit configured to perform image affine transformation on the first video frame to obtain a first transformed frame of the first video frame at the time when the intermediate frame is captured, and perform image affine transformation on the second video frame to obtain a second transformed frame of the second video frame at the time when the intermediate frame is captured; a prediction unit configured to predict a fusion weight of the first transformed frame and the second transformed frame based on the down-sampled image corresponding to the first video frame, the down-sampled image corresponding to the second video frame, the first transformed frame, the second transformed frame, the third optical flow and the fourth optical flow by using a preset neural network; a fusion unit configured to fuse pixels in the first transformed frame and the second transformed frame based on the fusion weight to obtain the intermediate frame, the fusion weight representing a probability that a pixel on the intermediate frame comes from the first video frame or from the second video frame.

10. An electronic device, comprising: The electronic device includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-8.

11. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method according to any one of claims 1-8.

Images