Video processing method and device, electronic device and storage medium

Abstract

The present disclosure provides a video processing method, apparatus, device and medium, and relates to the field of image and video processing technology. The implementation scheme is: determining a first frame and a second frame in a video; determining a motion vector between the first frame and the second frame based on the encoding data of the first frame and the second frame; determining an insertion time of a target synthetic frame for inserting between the first frame and the second frame based on the motion vector information; determining the target synthetic frame based on the first frame, the second frame and the insertion time; and inserting the target synthetic frame between the first frame and the second frame based on the insertion time.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image and video processing, and in particular to a video processing method, a video processing device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] By adding several frames between two consecutive frames in the video sequence, the display time between video frames can be shortened, thereby improving the frame rate and smoothness of the video.

[0003] The current interpolation methods include the following three: 1. Interpolation is performed by copying the previous frame to the next frame; 2. The intermediate frame is obtained by blurring the previous and next frames similar to double exposure; 3. The optical flow is generated by analyzing and modeling the previous and next frames to obtain a linear mapping relationship between frames, and finally the intermediate frame is combined.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or used. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The present disclosure provides a video processing method, a video processing device, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of the present disclosure, a video processing method is provided, including: determining a first frame and a second frame that are consecutive in a video; determining a motion vector between the first frame and the second frame based on encoded data of the first frame and the second frame; determining an interpolation moment for inserting a target synthetic frame between the first frame and the second frame based on the motion vector information; determining the target synthetic frame based on the first frame, the second frame and the interpolation moment; and inserting the target synthetic frame between the first frame and the second frame based on the interpolation moment.

[0007] According to another aspect of the present disclosure, a video processing device is also provided, including: a video frame determination unit, configured to determine a first frame and a second frame that are consecutive in a video; a motion vector determination unit, configured to determine a motion vector between the first frame and the second frame based on the encoding data of the first frame and the second frame; an interpolation moment determination unit, configured to determine an interpolation moment of a target synthetic frame for inserting between the first frame and the second frame based on the motion vector information; a synthesis unit, configured to determine the target synthetic frame based on the first frame, the second frame and the interpolation moment; and an interpolation unit, configured to insert the target synthetic frame between the first frame and the second frame based on the interpolation moment.

[0008] According to another aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, and the computer program implements the above method when executed by the at least one processor.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the above method when executed by a processor.

[0010] According to another aspect of the present disclosure, a computer program product is further provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0011] By utilizing the embodiments provided by the present disclosure, the interpolation time between two frames for interpolation can be determined based on the motion vector between the two frames in the video, thereby improving the problem of artifacts appearing in the synthesized frame due to large motion between the two frames and improving the visual effect of the video after interpolation.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1 An exemplary flow chart of a video processing method according to an embodiment of the present disclosure is shown;

[0015] Figure 2 A schematic illustration of motion estimation is shown;

[0016] Figure 3 A schematic illustration of motion compensation is shown;

[0017] Figure 4A FIG. 4C shows an exemplary illustration of a video frame insertion effect;

[0018] Figure 5 An exemplary block diagram of a video processing device according to an embodiment of the present disclosure is shown; and

[0019] Figure 6 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0021] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0022] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0023] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0024] By adding a number of frames to every two frames of the original video, the display time between video frames can be shortened, thereby increasing the frame rate and smoothness of the video and improving the viewing effect of the video.

[0025] Compared with the interpolation method of obtaining a synthesized intermediate frame by copying the picture of the previous frame or blurring the two frames before and after, the interpolation method based on the deep learning optical flow model can effectively model the mapping relationship between the target intermediate frame and the two frames before and after, so as to generate the intermediate frame in a more reasonable way. However, when there is a large movement between the original two frames before and after, there may be an inaccurate optical flow estimation, which will cause serious artifacts in the intermediate frame synthesized based on the optical flow model. For example, when there is an arm with a large swing amplitude in the picture, the synthesized moving arm often appears as a "broken limb". For another example, when there is a fast movement of the foreground / background in the picture, the synthesized foreground / background often appears blurred. Taking 2x interpolation as an example, the interpolation time is generally determined to be the middle position of the two frames. Due to the long time interval between the two frames before and after, the optical flow generated by the model at this position is relatively inaccurate. Such interpolation results have an adverse effect on the perception of the video after interpolation.

[0026] In order to solve the above problems, the present disclosure provides a video processing method.

[0027] Figure 1 An exemplary flow chart of a video processing method according to an embodiment of the present disclosure is shown.

[0028] like Figure 1 In step S102, a first frame and a second frame that are consecutive in a video are determined. In step S104, a motion vector between the first frame and the second frame is determined based on the encoded data of the first frame and the second frame. In step S106, an interpolation time of a target synthesized frame to be inserted between the first frame and the second frame is determined based on the motion vector information. In step S108, a target synthesized frame is determined based on the first frame, the second frame, and the interpolation time. In step S110, the target synthesized frame is inserted between the first frame and the second frame based on the interpolation time.

[0029] By using the video processing method provided by the present disclosure, the interpolation time for interpolation between the two frames can be determined based on the motion vector between the two frames, thereby improving the problem of artifacts appearing in the synthesized frame due to large motion between the two frames, and improving the visual effect of the video after interpolation. The coded data generated in the encoding and decoding process between the two frames already contains the motion vector between the two frames, so almost no additional resources and time loss are required, and only a simple calculation is required to obtain the motion vector between the two frames, thereby further determining the appropriate interpolation time between the two frames based on the motion vector information between the two frames.

[0030] Hereinafter, embodiments of the present disclosure will be described in detail.

[0031] In step S102, a first frame and a second frame that are continuous in the video may be determined. In some embodiments, the first frame and the second frame are from a continuous time-sequence video segment that belongs to the same shot.

[0032] Based on different encoding methods, video frames in a video can be divided into intra-coded frames (I frames), predicted frames (P frames) and bi-directional predicted frames (B frames). Among them, I frame images use intra-frame encoding, which only uses the spatial correlation within a single frame image for encoding without considering temporal correlation. Among these video frames, I frames do not depend on other frames and can be used as reference frames for decoding other frames. In contrast, P frames and B frames use inter-frame encoding, which simultaneously utilizes the spatial and temporal correlation of images.

[0033] When the latter of the two frames to be inserted is not an I frame, it means that there is an inter-frame reference between the two frames to be inserted during decoding, and therefore the motion vector information of the inter-frame reference needs to be calculated during the decoding process. In this case, the motion vector information between the two frames can be directly used to implement the method provided by the present disclosure and determine the insertion time of the synthetic frame to be inserted between the two frames. When the latter of the two frames to be inserted is an I frame, it means that there may be no inter-frame reference between the two frames to be inserted. For example, when changes such as transitions occur in the video, there may be no temporal or spatial correlation between the latter of the two frames to be inserted and the previous frame, so in this case, it may be meaningless to determine the insertion time based on the motion vector between the two frames.

[0034] Therefore, in some embodiments, when the latter of the two frames to be inserted is an I frame, the insertion time of the synthetic frame to be inserted can be directly determined as the middle time of the two frames (or any other suitable time). However, it can be understood that the embodiments of the present disclosure are not limited to this. In appropriate cases, even if the latter of the two frames to be inserted is an I frame, the video processing method provided by the present disclosure can also be applied to perform frame insertion.

[0035] In the embodiment of the present disclosure, the time position between two frames is represented by an interval of 0 to 1, 0 corresponds to the time of the first frame of the two frames to be interpolated, and 1 corresponds to the time of the second frame of the two frames to be interpolated. Based on this representation method, the middle time between two frames can be represented as 0.5.

[0036] In step S104 , a motion vector between the first frame and the second frame may be determined based on the encoded data of the first frame and the second frame.

[0037] Video coding can delete redundant and unimportant data in the video, thereby compressing the video to reduce the burden of physical storage and / or network transmission of the video. Typical video coding can be composed of three modules: predictive coding, transform coding, and entropy coding. Among them, predictive coding reduces the redundancy of the video by utilizing temporal (inter-frame prediction) and spatial redundancy (intra-frame prediction).

[0038] Predictive coding may include motion estimation and motion compensation. Figure 2 shows a schematic illustration of motion estimation, Figure 3 Schematic illustration of motion compensation is shown. Figure 2 As shown, motion estimation can find a suitable matching area 2021 in the reference frame 202 for the area (also called macroblock) 2011 of the current frame 201. The reference frame can be the previous frame of the current frame or the next frame of the current frame. Motion compensation can find the difference between area 2011 and area 2022. Predictive coding through motion estimation and motion compensation will generate some motion vectors. The motion vector indicates the motion trajectory of each area in the current frame relative to the corresponding matching area in the reference frame. Figure 3 As shown, the motion vector may indicate a motion trajectory of the region 3011 in the current frame 301 relative to the matching region 3021 in the reference frame 302 .

[0039] By determining the motion intensity between the first frame and the second frame based on the motion vector determined based on the encoded data, the method of the present disclosure does not require additional resources and time consumption.

[0040] Return to reference Figure 1 In step S106, an interpolation time of a target synthesized frame inserted between the first frame and the second frame may be determined based on the motion vector information.

[0041] In some embodiments, step S106 may include: determining a global motion parameter between the first frame and the second frame based on motion vector information between the first frame and the second frame, wherein the global motion parameter indicates a global motion intensity of the first frame and the second frame, and determining the interpolation moment based on the global motion parameter.

[0042] The motion vector information may include multiple motion vectors between multiple macroblocks in the first frame and multiple corresponding macroblocks in the second frame. Each motion vector indicates the motion trajectory between a macroblock in the first frame and its corresponding macroblock in the second frame. The maximum scalar value among the multiple motion vectors may be determined as a global motion parameter. In this case, the motion trajectory of the macroblock with the longest motion trajectory in the video frame may be used as a global motion parameter for representing the global motion intensity of the video frame. Using this method, it can be ensured that the final interpolation moment is also suitable for the area with the largest motion intensity in the video frame, and therefore also applies to other areas with smaller motion intensity in the video frame. In some examples, the scalar values ​​of multiple motion vectors may be screened, and after excluding the abnormal maximum values ​​therein, the maximum scalar value among the remaining motion vectors is determined as the global motion parameter. The specific determination method of the maximum scalar value of the motion vector is not limited here.

[0043] In some other examples, determining the global motion parameter between the first frame and the second frame based on the motion vector information may include:

[0044] For each macroblock in a video frame, determine the horizontal component of the motion vector of the macroblock in the horizontal direction and the vertical component of the motion vector in the vertical direction; determine the average motion horizontal component based on the horizontal component of the motion vector of each macroblock in the video frame, and determine the average motion vertical component based on the vertical component of the motion vector of each pixel in the video frame; based on the horizontal component, the vertical component, the average motion horizontal component and the average motion vertical component, determine the motion intensity index between the first frame and the second frame as a global motion parameter.

[0045] In practical applications, the global motion parameter between the first video frame and the second video frame can be determined by the following formula (1):

[0046] Ind=max(max(MV x ) / max(1.0,mean(abs(MV x ))), max(MV y ) /

[0047] max(1.0,mean(abs(MV y ))))(1)

[0048] Among them, Ind represents the global motion parameter; max is the maximum value function, min is the minimum value function, mean is the average value function, and abs is the absolute value function. MV x is the horizontal component of the macroblock’s motion vector MV in the horizontal direction, MV yis the vertical component of the motion vector MV of the macroblock in the vertical direction; max represents the maximum value. In some examples, the 99th percentile value of the motion vector of each macroblock can be taken as the maximum motion vector value. The purpose is to eliminate the influence of abnormally large values ​​on the final result. In other examples, the maximum scalar value of the motion vector of each macroblock can be directly used as the maximum motion vector value.

[0049] From the above formula (1), we can know that we can calculate the ratio of the maximum motion vector value to the average motion vector value on the two components respectively, and then take the largest ratio of the two components as the final global motion parameter. The global motion parameter is determined by using the average motion amplitude instead of directly using the maximum value of the motion vector as the final global motion parameter. This can reduce the problem that motion vectors of the same scale have different effects in video frames of different sizes (such as 1080P video frames and 540P video frames). In addition, mean(abs(MVx)) or mean(abs(MV y ) may be less than 1, so the lower limit of the average motion vector value is set to 1 in the above formula (1). This can prevent the global motion parameter result from being amplified when the average motion vector value is less than 1, and also prevent abnormal small values ​​from affecting the result.

[0050] like Figure 2 , 3 As shown, the motion vector can indicate the magnitude of the motion intensity of a region in a video frame relative to a reference frame. Therefore, based on the motion vectors of each region in a video frame relative to a corresponding region in a reference frame, a global motion parameter of the video frame can be determined. The global motion parameter can reflect whether there is a large motion between the first frame and the second frame, and the interpolation time can be determined based on the global motion parameter.

[0051] The size of the motion vector can be represented in various ways. In some examples, the size of the motion vector can be represented based on the number of pixels. For example, the size of the motion vector can be represented based on the number of pixels corresponding to the length of the motion trajectory of the macroblock between the first frame and the second frame. In other examples, the size of the motion vector can also be represented based on the ratio of the length of the motion trajectory to the size of the video frame. For example, the size of the motion vector can be represented based on the ratio of the length of the motion trajectory to a reference length. The reference length can be the length of the length, width, diagonal, or any other suitable size feature of the video frame.

[0052] In some embodiments, the interpolation time corresponding to the global motion parameter may be determined based on a predetermined correspondence. In some examples, the mapping relationship between the interpolation time corresponding to different global motion parameters may be predetermined. For example, the mapping relationship between the global motion parameter and the interpolation time may be represented by a table or a function. In other examples, the function indicating the mapping relationship between the global motion parameter and the interpolation time may be determined by fitting.

[0053] In some implementations, the greater the motion intensity indicated by the global motion parameter, the closer the interpolation moment is to one of the first frame and the second frame. In some examples, the greater the value of the global motion parameter, the greater the motion intensity it indicates. Also in the case where there is a large motion between the first frame and the second frame to be interpolated, the interpolation moment can be determined to be closer to one of the first frame and the second frame, rather than the middle moment of the first frame and the second frame. In the correspondence between the global motion parameter and the interpolation moment, one global motion parameter can correspond to two interpolation moments, one interpolation moment is offset forward close to the first frame, and the other interpolation moment is offset backward close to the second frame, and the two interpolation moments are symmetrical relative to the middle moment of the first frame and the second frame. When interpolating the moment based on the correspondence between the global motion parameter and the interpolation moment, the interpolation moment offset forward or the interpolation moment offset backward can be arbitrarily selected. In some examples, in the same video to be interpolated, the selection rules for selecting the interpolation moment between any two frames should be consistent. That is, in the same video to be interpolated, each interpolation moment is close to the first frame, or each interpolation moment is close to the second frame.

[0054] An example of the correspondence between the global motion parameters and the interpolation time is shown in the following table. The values ​​of the global motion parameters in the table are determined based on the size of the motion vector represented by the number of pixels in the video frame of size 1080P. When the video processing method according to the present disclosure is applied to video frames of other sizes, the global motion parameters in the following table can be adaptively scaled proportionally to obtain results suitable for video frames of other sizes.

[0055] Table 1

[0056] Global Motion Parameters Insertion moment Less than or equal to 15 0.5 Greater than 15 and less than or equal to 20 0.4 (or 0.6) Greater than 20 and less than or equal to 25 0.3 (or 0.7) Greater than 25 0.25 (or 0.75)

[0057] It can be understood that the mapping relationship given in Table 1 is only an exemplary description, and those skilled in the art can determine the specific values ​​of the mapping relationship between the global motion parameters and the interpolation time according to actual conditions.

[0058] In step S108 , a target synthesis frame is determined based on the first frame, the second frame and the interpolation time.

[0059] In some embodiments, the first frame and the second frame may be processed based on the interpolation time determined in step S106 using an interpolation model to obtain a target composite frame. The interpolation model may be a deep learning model based on optical flow (such as a real-time intermediate flow estimation algorithm, Real-Time Intermediate Flow Estimation, RIFE), or any other suitable model capable of implementing interpolation at any time.

[0060] In practical applications, two consecutive video frames and the target interpolation time required for interpolation are input into the interpolation model. The optical flow analysis layer of the interpolation model will first generate a first optical flow corresponding to the first timestamp of the previous frame to the target interpolation time and a second optical flow from the target interpolation time to the second timestamp of the next frame. Then, the mapping operation (warp operation) of the layer is adopted to sample from the two input video frames respectively to obtain the first sampling result and the second sampling result. The fusion layer of the interpolation model is used to fuse the two sampling results according to the fusion weights generated by the interpolation model to generate the target synthetic frame corresponding to the final target interpolation time.

[0061] It should be noted that the movement direction and size of a certain pixel point in the first video frame between the two video frames can be known from the obtained optical flow, and then the corresponding pixel point can be mapped back to the corresponding coordinate position according to the two obtained optical flows through a mapping operation (warp operation), so as to realize sampling of the two input video frames. In addition, setting the fusion weight is an intermediate output result of the interpolation model, and the interpolation model can calculate the weight of each pixel point of the two video frames before and after by itself. For example, pixel point a of the first frame and pixel point b of the second frame are both mapped to a certain position C of the middle synthetic frame, but only one pixel point can be placed in one position. At this time, it is necessary to weigh the proportion of the two pixel points a and b in C. At this time, the above-mentioned setting of fusion weights can be used to balance.

[0062] In an embodiment of the present application, the motion vector information can be used as a priori information to adaptively guide the interpolation time of the interpolation model to move forward or backward. Moving forward, the target synthetic frame will be closer to the previous frame. At this time, the optical flow from the first timestamp to the target interpolation time will be relatively small. Although the optical flow from the target interpolation time to the second timestamp will be large, even if the optical flow estimation from the target interpolation time to the second timestamp is inaccurate, during the fusion process, due to the interpolation model mechanism, the fusion weight of the optical flow from the target interpolation time to the second timestamp will be very small, so it will not have a substantial impact on the final result. Ultimately, the smaller fusion weight will reduce the impact of the inaccurate optical flow from the target interpolation time to the second timestamp on the result, ensuring that the target synthetic frame will not be similar to the previous and next frames, but will be closer to the first video frame. Similarly, if the interpolation time of the target synthetic frame is moved backward, the target synthetic frame can be made more similar to the next frame. In this way, by utilizing the characteristics that the closer the interpolated frame is to any input frame at any time, the fewer the interpolated frame artifacts are, the method provided by the present disclosure can greatly improve the problem of artifacts appearing in the interpolated frame at the intermediate moment caused by large motion.

[0063] In step S110 , a target synthesized frame is inserted between the first frame and the second frame based on the frame insertion time.

[0064] In the embodiments of the present disclosure, the principle of the present disclosure is described by taking 2x interpolation (i.e., inserting one frame between every two frames) as an example. However, the scope of the present disclosure is not limited thereto. Those skilled in the art may apply the method for determining the interpolation time based on motion vector information provided by the present disclosure in other interpolation processes according to actual conditions, so as to determine the appropriate interpolation time based on the image content of the video, thereby improving the viewing experience of the video.

[0065] Figure 4A FIG4C shows an exemplary description of the video frame insertion effect. Figure 4ACorresponding to the first frame, FIG4B corresponds to the second frame. It can be seen that between the first frame and the second frame, there is a large movement of the arm of the character in the picture. If the interpolation is performed according to the conventional interpolation method, that is, the interpolation is performed at the middle moment of the first frame and the second frame, the synthetic frame generated by the interpolation model based on the optical flow has a large optical flow between the previous frame and the next frame, so the generated synthetic frame may be as shown in FIG4C, and the arm of the character in the picture is no longer complete. However, if the video processing method provided by the present disclosure is adopted, the interpolation moment can be determined to be closer to the first frame or closer to the second frame. In the case where the interpolation moment is closer to the first frame, the picture of the synthetic frame generated by the interpolation model is closer to the first frame (that is, the arm is raised higher). In the case where the interpolation moment is closer to the second frame, the picture of the synthetic frame generated by the interpolation model is closer to the second frame (that is, the arm is lowered lower). In this case, since the optical flow between the synthetic frame and the adjacent video frame is small, the model predicts the synthetic frame more accurately and the picture effect will be better. In this case, by adjusting the interpolation moment between the two frames based on the size of the motion intensity between the two frames, the viewing effect of the video after interpolation can be improved.

[0066] Figure 5 An exemplary block diagram of a video processing device according to an embodiment of the present disclosure is shown.

[0067] like Figure 5 As shown, the video processing device 500 may include a video frame determination unit 510 , a motion vector determination unit 520 , a frame insertion time determination unit 530 , a synthesis unit 540 and a frame insertion unit 550 .

[0068] The video frame determination unit 510 may be configured to determine a first frame and a second frame that are consecutive in the video. The motion vector determination unit 520 may be configured to determine a motion vector between the first frame and the second frame based on the encoded data of the first frame and the second frame. The interpolation time determination unit 530 may be configured to determine an interpolation time for inserting a target synthesized frame between the first frame and the second frame based on the motion vector information. The synthesis unit 540 may be configured to determine a target synthesized frame based on the first frame, the second frame, and the interpolation time. The interpolation unit 550 may be configured to insert the target synthesized frame between the first frame and the second frame based on the interpolation time.

[0069] By using the video processing method provided by the present disclosure, the interpolation time for interpolation between the two frames can be determined based on the motion vector between the two frames, thereby improving the problem of artifacts appearing in the synthesized frame due to large motion between the two frames, and improving the visual effect of the video after interpolation. The coded data generated in the encoding and decoding process between the two frames already contains the motion vector between the two frames, so almost no additional resources and time loss are required, and only a simple calculation is required to obtain the motion vector between the two frames, thereby further determining the appropriate interpolation time between the two frames based on the motion vector information between the two frames.

[0070] It should be understood that Figure 5 The various units of the device 500 shown in FIG. 5 can be used in conjunction with the reference Figure 1 The steps in the method 100 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 100 are also applicable to the device 800 and the units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0071] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 5 The various units described may be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units may be implemented as computer program code / instructions configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units may be implemented as hardware logic / circuits. For example, in some embodiments, one or more of the video frame determination unit 510, the motion vector determination unit 520, the interpolation moment determination unit 530, the synthesis unit 540, and the interpolation unit 550 may be implemented together in a system on chip (SoC). The SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and may optionally execute the received program code and / or include embedded firmware to perform functions.

[0072] According to another aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, and the computer program implements the above method when executed by the at least one processor.

[0073] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the above method when executed by a processor.

[0074] According to another aspect of the present disclosure, a computer program product is further provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0075] See also Figure 6 , a block diagram of an electronic device 600 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device can be a computer device of different types, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0076] like Figure 6 As shown, the electronic device 600 may include at least one processor 601 , a working memory 602 , an input unit 604 , a display unit 605 , a speaker 606 , a storage unit 607 , a communication unit 608 , and other output units 606 that can communicate with each other via a system bus 603 .

[0077] The processor 601 may be a single processing unit or multiple processing units, all of which may include a single or multiple computing units or multiple cores. The processor 601 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. The processor 601 may be configured to obtain and execute computer-readable instructions stored in a working memory 602, a storage unit 607, or other computer-readable media, such as program codes of an operating system 602a, program codes of an application 602b, and the like.

[0078] The working memory 602 and the storage unit 607 are examples of computer-readable storage media for storing instructions that are executed by the processor 601 to implement the various functions described above. The working memory 602 may include both volatile memory and non-volatile memory (e.g., RAM, ROM, etc.). In addition, the storage unit 607 may include a hard disk drive, a solid-state drive, a removable medium, including external and removable drives, a memory card, a flash memory, a floppy disk, an optical disk (e.g., a CD, a DVD), a storage array, a network attached storage, a storage area network, etc. The working memory 602 and the storage unit 607 may all be collectively referred to herein as memory or computer-readable storage media, and may be a non-transitory medium capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by the processor 601 as a specific machine configured to implement the operations and functions described in the examples herein.

[0079] The input unit 606 may be any type of device capable of inputting information to the electronic device 600. The input unit 606 may receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and may include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone and / or a remote controller. The output unit may be any type of device capable of presenting information, and may include but is not limited to a display unit 605, a speaker 606, and other output units 609, which may include but are not limited to video / audio output terminals, vibrators and / or printers. The communication unit 608 allows the electronic device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth device, an 802.6 device, a WiFi device, a WiMax device, a cellular communication device and / or the like.

[0080] The application 602b in the working register 602 can be loaded to execute the various methods and processes described above, for example Figure 1. For example, in some embodiments, the method 100 described above may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 607. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the storage unit 607 and / or the communication unit 608. When the computer program is loaded and executed by the processor 601, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, the processor 601 may be configured to perform the method 100 in any other appropriate manner (e.g., by means of firmware).

[0081] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0082] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0083] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0084] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0085] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0086] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0087] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0088] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A video processing method, comprising: Determine the first and second consecutive frames in the video; determining a motion vector between the first frame and the second frame based on the encoded data of the first frame and the second frame; Determine, based on the motion vector information, a frame insertion time of a target synthesized frame inserted between the first frame and the second frame; Determine the target synthesized frame based on the first frame, the second frame and the interpolation time; as well as inserting the target synthesized frame between the first frame and the second frame based on the interpolation time, Wherein, determining the interpolation time of a target synthesized frame for inserting between the first frame and the second frame based on the motion vector information includes: determining a global motion parameter between the first frame and the second frame based on the motion vector information, wherein the global motion parameter indicates a global motion strength of the first frame and the second frame; The interpolation frame time is determined based on the global motion parameter, wherein the greater the motion intensity indicated by the global motion parameter is, the closer the interpolation frame time is to one of the first frame and the second frame.

2. The video processing method according to claim 1, wherein: The motion vector information includes a plurality of motion vectors between a plurality of macroblocks in the first frame and a plurality of corresponding macroblocks in the second frame, Determining a global motion parameter between the first frame and the second frame based on the motion vector information includes: A maximum scalar value among the plurality of motion vectors is determined as the global motion parameter.

3. The video processing method according to claim 1, wherein: The motion vector information includes a plurality of motion vectors between a plurality of macroblocks in the first frame and a plurality of corresponding macroblocks in the second frame, Determining a global motion parameter between the first frame and the second frame based on the motion vector information includes: For each macroblock in the video frame, determining a horizontal axis component of a motion vector of a pixel point in the macroblock in the horizontal axis direction and a vertical axis component of the motion vector in the vertical axis direction; Determine an average motion horizontal axis component based on the horizontal axis component of the motion vector of each macroblock in the video frame, and determine an average motion vertical axis component based on the vertical axis component of the motion vector of each macroblock in the video frame; Based on the horizontal axis component, the vertical axis component, the average motion horizontal axis component and the average motion vertical axis component, a motion intensity index between the first frame and the second frame is determined as the global motion parameter.

4. The video processing method according to claim 1, wherein: Determining the interpolation frame time based on the global motion parameter comprises: An interpolation frame time corresponding to the global motion parameter is determined based on a predetermined corresponding relationship.

5. A video processing device, comprising: A video frame determination unit, configured to determine a first frame and a second frame that are consecutive in the video; a motion vector determination unit configured to determine a motion vector between the first frame and the second frame based on the encoded data of the first frame and the second frame; a frame insertion time determination unit configured to determine a frame insertion time of a target synthesized frame inserted between the first frame and the second frame based on the motion vector information; A synthesis unit, configured to determine the target synthesis frame based on the first frame, the second frame and the interpolation time; as well as a frame insertion unit, configured to insert the target synthesized frame between the first frame and the second frame based on the frame insertion time, Wherein, determining the interpolation time of a target synthesized frame for inserting between the first frame and the second frame based on the motion vector information includes: determining a global motion parameter between the first frame and the second frame based on the motion vector information, wherein the global motion parameter indicates a global motion strength of the first frame and the second frame; The interpolation frame time is determined based on the global motion parameter, wherein the greater the motion intensity indicated by the global motion parameter is, the closer the interpolation frame time is to one of the first frame and the second frame.

6. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores a computer program, which, when executed by the at least one processor, implements the method according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium storing a computer program, wherein: The computer program implements the method according to any one of claims 1 to 4 when executed by a processor.

8. A computer program product comprising a computer program, wherein: The computer program implements the method according to any one of claims 1 to 4 when executed by a processor.

Images