Video processing method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides a video processing method and device, electronic equipment and storage medium, relating to the technical field of image and video processing. The implementation scheme is: determining a first frame and a second frame in a video; determining motion intensity between the first frame and the second frame based on motion information between the first frame and the second frame; determining an inter-frame optical flow map between the first frame and the second frame based on the motion intensity; and determining a target synthesis frame based on the first frame, the second frame and the inter-frame optical flow map.

Description

Technical Field

[0001] This disclosure relates to the field of image and video processing technology, specifically to a video processing method, a video processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] By adding several frames between two consecutive frames in a video sequence, the display time between video frames can be shortened, thereby improving the video's frame rate and smoothness.

[0003] The current frame interpolation methods include the following three types: 1. Interpolating the frame by copying the previous frame to the next frame; 2. Obtaining the intermediate frame by performing a blurring process similar to double exposure on the two frames; 3. Generating optical flow by analyzing and modeling the two frames to obtain the inter-frame linear mapping relationship, and finally combining them to obtain the intermediate frame.

[0004] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0005] This disclosure provides a video processing method, a video processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of this disclosure, a video processing method is provided, comprising: determining a first frame and a second frame consecutively in a video; determining a motion intensity between the first frame and the second frame based on motion information between the first frame and the second frame; determining an inter-frame optical flow graph for determining the first frame and the second frame based on the motion intensity; and determining a target composite frame based on the first frame, the second frame, and the inter-frame optical flow graph.

[0007] According to another aspect of this disclosure, a video processing apparatus is also provided, comprising: a video frame determination unit configured to determine a first frame and a second frame consecutively in a video; a motion intensity determination unit configured to determine a motion intensity between the first frame and the second frame based on motion information between the first frame and the second frame; an optical flow estimation unit configured to determine an inter-frame optical flow map for determining the first frame and the second frame based on the motion intensity; and a synthesis unit configured to determine a target synthesized frame based on the first frame, the second frame, and the inter-frame optical flow map.

[0008] According to another aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein said memory stores a computer program that, when executed by said at least one processor, implements the method described above.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the method described above when executed by a processor.

[0010] According to another aspect of this disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the method described above.

[0011] Using the embodiments provided in this disclosure, the inter-frame optical flow map between two frames can be adaptively determined based on the intensity between the two frames, thereby enabling the optical flow estimation results to be adaptively adjusted for different motion amplitudes present in the image, thereby improving the overall optical flow accuracy and enhancing the sensory effect of the synthesized frame.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1 An exemplary flowchart of a video processing method according to an embodiment of the present disclosure is shown;

[0015] Figure 2 A schematic illustration of motion compensation is shown;

[0016] Figure 3 An exemplary process of a video processing method according to an embodiment of the present disclosure is shown;

[0017] Figures 4A to 4C An exemplary illustration of video frame interpolation effect is shown;

[0018] Figure 5 An exemplary block diagram of a video processing apparatus according to embodiments of the present disclosure is shown; and

[0019] Figure 6A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0022] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0023] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0024] By adding several frames between every two frames displayed in the original video, the display time between video frames can be shortened, thereby improving the video's frame rate and smoothness, and enhancing the viewing experience.

[0025] Compared to frame interpolation methods that involve copying the previous frame or blurring the preceding and following frames to obtain the synthesized intermediate frame, frame interpolation based on a deep learning optical flow model can effectively model the mapping relationship between the target intermediate frame and the preceding and following frames, thus generating intermediate frames in a more reasonable way. However, when there is significant motion between the original preceding and following frames, inaccurate optical flow estimation may occur, leading to severe artifacts in the intermediate frame synthesized based on the optical flow model. For example, when there is an arm with a large swinging amplitude in the image, the synthesized moving arm often appears as a "severed limb." Similarly, when there is rapid movement of the foreground / background in the image, the synthesized foreground / background often appears blurred.

[0026] To address the aforementioned problems, this disclosure provides a video processing method.

[0027] Figure 1 An exemplary flowchart of a video processing method according to an embodiment of the present disclosure is shown.

[0028] like Figure 1 In step S102, a first frame and a second frame consecutively in the video are determined. In step S104, the motion intensity between the first frame and the second frame is determined based on the motion information between them. In step S106, an inter-frame optical flow graph is determined based on the motion intensity to determine the inter-frame optical flow graph between the first frame and the second frame. In step S108, the target composite frame is determined based on the first frame, the second frame, and the inter-frame optical flow graph.

[0029] Using the video processing method provided in this disclosure, the inter-frame optical flow map between two frames can be adaptively determined based on the intensity between the two frames, thereby enabling the optical flow estimation results to be adaptively adjusted for different motion amplitudes present in the image, thereby improving the overall optical flow accuracy and enhancing the sensory effect of the synthesized frame.

[0030] The embodiments of this disclosure will now be described in detail.

[0031] In step S102, a first frame and a second frame consecutively in the video can be determined. In some embodiments, the first frame and the second frame come from a consecutive temporal video segment belonging to the same shot.

[0032] In step S104, the motion intensity between the first frame and the second frame can be determined based on the motion information between the first frame and the second frame.

[0033] Motion information can indicate the differences between the first and second frames, thus reflecting the motion scale between the first and second frames.

[0034] In some embodiments, motion information may include motion vectors or inter-frame optical flow between the first frame and the second frame. The motion vectors or inter-frame optical flow can reflect the offset between an object in the first frame and its corresponding object in the second frame, thereby reflecting the intensity of the object's motion from the first frame to the second frame. A greater offset from the first frame to the second frame indicates a greater motion intensity.

[0035] Motion vectors between the first and second frames can be determined based on video coding data. Typical video coding consists of three modules: predictive coding, transformative coding, and entropy coding. Predictive coding reduces video redundancy by utilizing temporal (inter-frame prediction) and spatial redundancy (intra-frame prediction). Predictive coding can include motion estimation and motion compensation. Predictive coding, through motion estimation and motion compensation, generates motion vectors. These motion vectors indicate the trajectory of each region in the current frame relative to its corresponding matching region in the reference frame. Figure 2 As shown, motion vectors can indicate the motion trajectory of region 2011 in the current frame 201 relative to the matching region 2021 in the reference frame 202. By determining the motion intensity between the first and second frames based on motion vectors determined from encoded data during the video encoding process, the method disclosed herein requires no additional resources or time consumption. Motion vector information can include multiple motion vectors between multiple macroblocks in the first frame and multiple corresponding macroblocks in the second frame. Each motion vector indicates the motion trajectory between a macroblock in the first frame and its corresponding macroblock in the second frame.

[0036] Inter-frame optical flow (IPF) between the first and second frames can be determined based on optical flow estimation. Optical flow is a motion vector representing the direction and distance of motion of a pixel; therefore, IPF indicates the motion information of each pixel from the first frame to the second frame, i.e., it indicates the changes in each pixel. The first and second frames can be processed using an optical flow estimation model to obtain the IPF map output by the model. This optical flow estimation model can be any deep learning-based model (e.g., RAFT (Recurrent All Pairs Field Transforms for Optical Flow); FlowNet (Learning Optical Flow with Convolutional Networks); RIFE (Real-Time Intermediate Flow Estimation) etc.) or a traditional optical flow estimation algorithm (e.g., Lucas-Kanade, a two-frame difference optical flow estimation algorithm). In some examples, IPF may refer to the optical flow from the first frame to the second frame. In other examples, inter-frame optical flow may refer to the optical flow from the first frame to the target interpolation time and / or the optical flow from the target interpolation time to the second frame.

[0037] In some embodiments, step S104 may include: determining the maximum motion index between the first frame and the second frame as a global motion parameter based on motion information. The global motion parameter indicates the motion intensity between the first frame and the second frame.

[0038] When motion information includes motion vectors, the maximum motion metric can refer to the maximum motion vector. In some examples, the maximum scalar value among multiple motion vectors can be determined as the global motion parameter. In this case, the motion trajectory of the macroblock with the longest motion trajectory in the video frame can be used as the global motion parameter to represent the global motion intensity of the video frame. In some examples, the scalar values ​​of multiple motion vectors can be filtered to exclude outliers, and the maximum scalar value among the remaining motion vectors can be determined as the global motion parameter. The specific method for determining the maximum scalar value of the motion vector is not limited here.

[0039] In other examples, the maximum motion metric can refer to the maximum relative motion vector. The maximum relative motion vector can be determined based on the horizontal, vertical, average horizontal, and average vertical components of the motion vectors of each macroblock in a video frame. For example, the maximum relative motion vector between the first and second frames can be determined as a global motion parameter using the following formula (1):

[0040] Ind = max(max(MV) x ) / max(1.0,mean(abs(MV) x ))), max(MV y ) / max(1.0,mean(abs(MV) y )))) (1)

[0042] Where Ind represents the global motion parameters; max is the maximum value function, min is the minimum value function, mean is the average value function, and abs is the absolute value function. MV x Let MV be the horizontal component of the macroblock's motion vector MV along the horizontal axis. y MV represents the vertical component of the macroblock's motion vector MV along the vertical axis; max represents the maximum value. In some examples, the output of the max function can be the 99th percentile (or other predefined percentile) value, the purpose of which is to exclude the influence of abnormally large values ​​on the final result. In other examples, the maximum value can be used directly as the output of the max function.

[0043] When motion information includes inter-frame optical flow, the maximum motion metric can refer to the maximum optical flow. In some examples, the maximum motion metric can be the maximum optical flow between the first timestamp corresponding to the first frame and the target interpolation time, or the maximum optical flow between the target interpolation time and the second timestamp corresponding to the second frame.

[0044] For example, the maximum optical flow between the first and second frames can be determined as a global motion parameter using the following formula (2).

[0045]

[0046] Where Ind represents the global motion parameters; max is the maximum value function, f x and f y Let f be the components of the optical flow f in the x and y directions, respectively. 0→t f represents the optical flow between the first timestamp 0 of the first frame and the target interpolation time t. t→1 This represents the optical flow between the target interpolation time t and the second timestamp 1 corresponding to the second frame. In some examples, the output of the max function can be the result of the 99th percentile (or other predefined percentile) value, the purpose of which is to exclude the influence of abnormally large values ​​on the final result. In other examples, the maximum value can be used directly as the output of the max function.

[0047] In other examples, the maximum motion metric can be the maximum relative optical flow between the first and second frames. The maximum relative optical flow can be determined based on the horizontal and vertical components of the optical flow at each pixel, the horizontal component of the average optical flow, and the vertical component of the average optical flow. For example, the maximum relative optical flow between the first and second frames can be determined as a global motion parameter using the following formula (3):

[0048] Ind=max(max(Fx) / max(1.0,mean(abs(F x ))), max(F y ) / max(1.0,mean(abs(F y )))) (3)

[0050] Where Ind represents the global motion parameters; max is the maximum value function, min is the minimum value function, mean is the average value function, and abs is the absolute value function. F x Let F be the horizontal component of the optical flow F of the target pixel in the horizontal direction. yrepresents the vertical component of the optical flow F of the target pixel along the vertical axis; max represents the maximum value. In some examples, the output of the max function can be the 99th percentile (or other predefined percentile) value, the purpose of which is to exclude the influence of abnormally large values ​​on the final result. In other examples, the maximum value can be used directly as the output of the max function.

[0051] In step S106, an inter-frame optical flow map for determining the first frame and the second frame can be determined based on motion intensity.

[0052] The greater the motion intensity, the larger the receptive field used to determine the inter-frame optical flow map. When there is large motion between two frames, the model's estimation of optical flow between the two frames is inaccurate. This is because when there is large motion, an object can span a large area in the image. When the receptive field used for optical flow estimation is insufficient, it will be unable to capture the connection between the object and the two frames well, resulting in biased optical flow results for the parts of the object with large motion. In some embodiments, when the motion intensity exceeds a predetermined threshold, the receptive field of optical flow can be increased.

[0053] In some implementations, the receptive field of the model can be increased by increasing the number of model parameters, thereby enhancing the model's global modeling ability and obtaining better optical flow estimation results. In other implementations, the receptive field can be relatively expanded by reducing the size of the original image frames. When the image size is reduced, the large motions of objects between frames are proportionally reduced to small motions. In this case, even without increasing the number of model parameters, it is possible to generate optical flow between two frames using a sufficient receptive field. Although smaller optical flow may lose some detail, it will better capture large motions between two frames, thus avoiding severe artifacts. Furthermore, using smaller optical flow can also improve inference speed and enhance the overall performance of the model.

[0054] When the image size is reduced, step S106 may include: determining a scaling factor for the inter-frame optical flow map based on motion intensity; scaling the first frame and the second frame based on the scaling factor to obtain a scaled first frame and a scaled second frame; performing optical flow estimation on the scaled first frame and the scaled second frame to obtain a first optical flow between a first timestamp corresponding to the first frame and the target interpolation time, and a second optical flow between a second timestamp corresponding to the second frame and the target interpolation time, wherein the dimensions of the first optical flow and the second optical flow are the same as the dimensions of the scaled first frame and the scaled second frame; and determining the inter-frame optical flow map based on the first optical flow and the second optical flow.

[0055] The first and second optical flows can be obtained using any suitable optical flow estimation model (such as IFNet). The specific method used for optical flow estimation is not limited here.

[0056] In some embodiments, a scaling factor corresponding to the motion intensity can be determined based on a predetermined mapping relationship.

[0057] In some implementations, when the motion intensity is greater than 25, the scaling factor can be set to a reduction of 2. When the motion intensity is not greater than 25, the scaling factor can be set to 1, meaning no scaling is performed. In other implementations, when the motion intensity is greater than 25, the scaling factor can be set to a reduction of 4. When the motion intensity is not greater than 25, the scaling factor can be set to 1, meaning no scaling is performed. In still other implementations, the original video size can be considered when determining the predetermined mapping relationship. For example, for 1080p video, when the motion intensity is greater than 25, the scaling factor can be set to a reduction of 2; for 4K video, the scaling factor can be set to a reduction of 4; when the motion intensity is not greater than 25, the scaling factor can be set to 1, meaning no scaling is performed.

[0058] It is understood that the above mapping relationship is merely an exemplary illustration, and those skilled in the art can modify the above example based on actual circumstances to obtain other mapping relationships for determining motion intensity and scaling factor.

[0059] By reducing the input image frame size for optical flow estimation when the motion intensity exceeds a threshold (i.e., large motion exists in the image), the model's receptive field for motion in the image can be increased without increasing the model's parameters, thereby enhancing its ability to model large motion. Furthermore, the smaller input image frame size also improves the model's running speed.

[0060] In step S108, the target composite frame can be determined based on the first frame, the second frame, and the inter-frame optical flow map. The target composite frame can be obtained using the image information in the first and second frames and the motion information in the inter-frame optical flow map.

[0061] The target composite frame can be obtained using any optical flow-based image fusion model. In some examples, the fusion layer of the RIFE model can be used to determine the target composite frame.

[0062] In some embodiments, when optical flow estimation is performed on the scaled first frame and the scaled second frame in step S106 to obtain the first optical flow and the second optical flow, the first optical flow and the second optical flow can be inversely scaled to restore the size of the optical flow map to the same size as the original first frame and the second frame to obtain the target composite frame, wherein the size of the target composite frame is the same as the first frame and the second frame.

[0063] In this case, step S108 may include: inverse scaling the first optical flow and the second optical flow based on a scaling factor to obtain the inversely scaled first optical flow and the inversely scaled second optical flow, wherein the size of the inversely scaled first optical flow and the inversely scaled second optical flow is the same as the size of the first frame and the second frame; and determining the target composite frame based on the inversely scaled first optical flow, the inversely scaled second optical flow, the first frame and the second frame.

[0064] The final target composite frame can be obtained by fusing the inversely scaled first optical flow with the first frame and the inversely scaled second optical flow with the second frame, based on these two fusion results. Optical flow and image frames can be fused in any possible way. In some implementations, a first sampling result can be obtained by sampling from the first frame based on the inversely scaled first optical flow, and a second sampling result can be obtained by sampling from the second frame based on the inversely scaled second optical flow. The first and second sampling results can then be fused to obtain the target composite frame for the target interpolation time.

[0065] When the scaling factor is reduced by a factor of 2, the scaling operation means reducing the image size to half of its current size, while the inverse scaling operation means enlarging the image size to twice its current size.

[0066] In other embodiments, if optical flow estimation is performed on the scaled first frame and the scaled second frame in step S106 to obtain the first optical flow and the second optical flow, image fusion can be performed based on the first optical flow, the second optical flow, and the scaled first frame and the scaled second frame to obtain a composite image with the same size as the first optical flow and the second optical flow. Then, the composite image can be inversely scaled based on the scaling factor to obtain a target composite frame with the same size as the first frame and the second frame.

[0067] The obtained target composite frame can be used to interpolate the video. Interpolation can be achieved by inserting the target composite frame into the target interpolation time between the first and second frames. In some implementations, the target interpolation time can be determined based on predetermined rules. For example, the target interpolation time can be determined in a uniformly distributed manner. For 2x interpolation, when the first timestamp corresponding to the first frame is 0 and the second timestamp corresponding to the second frame is 1, the target interpolation time can be 0.5. Alternatively, the specific value of the target interpolation time can be predetermined, such as any position between 0 and 1. In other implementations, the target interpolation time can be determined based on the motion intensity obtained in step S104. The greater the motion intensity, the closer the target interpolation time is to either the first or second frame. Determining the target interpolation time in this way makes the obtained target composite frame closer to either the first or second frame (i.e., closer to the target interpolation time), thereby further improving the visual effect of interpolation when there is large motion.

[0068] Figure 3 An exemplary process of a video processing method according to an embodiment of the present disclosure is shown.

[0069] like Figure 3 As shown, the input to the video processing method can be determined at box 301, where the input image frames may include the first and second consecutive frames in the video.

[0070] At box 302, the inter-frame optical flow graph between the two input frames can be determined. The inter-frame optical flow graph can include the optical flow from the first timestamp corresponding to the first frame to the target interpolation time, and the optical flow from the target interpolation time to the second timestamp corresponding to the second frame.

[0071] At box 303, it can be determined whether there is large motion between the two input frames.

[0072] If the output of box 303 is "No (N)", meaning there is no large motion between the two frames, the target composite frame can be determined directly at box 304 based on the optical flow determined at box 302 and the input image determined at box 301.

[0073] If the output of box 303 is "Yes (Y)", meaning there is a large motion between the two frames, then a scaling operation can be performed on the two input frames at box 305 to obtain a scaled (smaller) image frame.

[0074] At box 306, the inter-frame optical flow map between two frames can be determined based on the scaled image frame obtained at box 305.

[0075] At box 307, the target composite frame can be determined based on the scaled image frame determined at box 301 and the inter-frame optical flow map determined at box 306. An optical flow of the same size as the input image frame at box 301 can be obtained by inverse scaling the inter-frame optical flow map determined at box 306, and then the inversely scaled optical flow is fused with the input image at box 301 to obtain the target composite frame of the same size as the input image frame at box 301.

[0076] The RIFE model can be used to achieve optical flow estimation and optical flow-based image fusion in the embodiments of this disclosure. When using the RIFE model for optical flow estimation, an inter-frame optical flow map between two frames can be determined, and a fusion map and a residual can be obtained. The fusion map is a single-channel mask with a value range in the (0, 1) interval and the same width and height as the input image; the residual is a three-channel residual map with a value range in the (0, 1) interval and the same width and height as the input image, used to supplement edge details in the fused interpolation result. When using the RIFE model to fuse optical flow and image, the input image frame can be backward-warped with an optical flow map of the same size to obtain a first sampling result and a second sampling result. The first sampling result, the second sampling result, the fusion map, and the residual are then fused to obtain the target composite frame.

[0077] It is understood that the RIFE model is merely one example for implementing embodiments of this disclosure. Any other suitable model may be used to implement the methods of this disclosure without departing from the principles of this disclosure.

[0078] Figures 4A to 4C An exemplary illustration of video frame interpolation effect is shown. Wherein Figure 4A Corresponding to the first frame, Figure 4B Corresponding to the second frame, it can be seen that there is significant movement of the character's arm between the first and second frames. If optical flow estimation is performed using conventional methods, the resulting composite frame might look like... Figure 4C As shown, the arm of the person in the image is no longer complete. However, if the video processing method provided in this disclosure is used, the scale of the receptive field used for optical flow estimation can be adaptively determined during optical flow estimation. Using the method provided in this disclosure, the optical flow estimation effect can be improved when there is large motion in the image, and thus the effect of optical flow-based target prediction composite frames can be improved.

[0079] Figure 5 An exemplary block diagram of a video processing apparatus according to an embodiment of the present disclosure is shown.

[0080] like Figure 5As shown, the video processing apparatus 500 may include a video frame determination unit 510, a motion intensity determination unit 520, an optical flow estimation unit 530, and a synthesis unit 540.

[0081] The video frame determination unit 510 can be configured to determine a first frame and a second frame consecutively in the video. The motion intensity determination unit 520 can be configured to determine the motion intensity between the first frame and the second frame based on motion information between them. The optical flow estimation unit 530 can be configured to determine an inter-frame optical flow map based on the motion intensity. The synthesis unit 540 can be configured to determine a target composite frame based on the first frame, the second frame, and the inter-frame optical flow map.

[0082] Using the video processing method provided in this disclosure, the inter-frame optical flow map between two frames can be adaptively determined based on the intensity between the two frames, thereby enabling the optical flow estimation results to be adaptively adjusted for different motion amplitudes present in the image, thereby improving the overall optical flow accuracy and enhancing the sensory effect of the synthesized frame.

[0083] It should be understood that Figure 5 Each unit of the device 500 shown can be connected to a reference. Figure 1 The steps in method 100 described correspond to each other. Therefore, the operations, features, and advantages described above for method 100 also apply to apparatus 500 and its constituent units. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0084] It should also be understood that this article can describe various technologies in the general context of software and hardware components or program modules. The above regarding... Figure 5The described units can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuit. For example, in some embodiments, one or more of the video frame determination unit 510, motion vector determination unit 520, frame interpolation timing determination unit 530, synthesis unit 540, and frame interpolation unit 550 can be implemented together in a System on Chip (SoC). The SoC may include an integrated circuit chip (which includes a processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or one or more components of other circuitry) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0085] According to another aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein said memory stores a computer program that, when executed by said at least one processor, implements the method described above.

[0086] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the method described above when executed by a processor.

[0087] According to another aspect of this disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the method described above.

[0088] See Figure 6 The present invention describes a structural block diagram of an electronic device 600 that can serve as a server or client of the present disclosure, which is an example of hardware devices that can be applied to various aspects of the present disclosure. The electronic device can be different types of computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0089] like Figure 6 As shown, the electronic device 600 may include at least one processor 601, working memory 602, input unit 604, display unit 605, speaker 606, storage unit 607, communication unit 608 and other output units 606 that are capable of communicating with each other via system bus 603.

[0090] Processor 601 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 601 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Processor 601 may be configured to acquire and execute computer-readable instructions stored in working memory 602, storage unit 607, or other computer-readable media, such as program code of operating system 602a, program code of application program 602b, etc.

[0091] Working memory 602 and storage unit 607 are examples of computer-readable storage media for storing instructions that are executed by processor 601 to perform the various functions described above. Working memory 602 may include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, storage unit 607 may include hard disk drives, solid-state drives, removable media including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Working memory 602 and storage unit 607 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code that can be executed by processor 601 as a specific machine configured to perform the operations and functions described in the examples herein.

[0092] Input unit 606 can be any type of device capable of inputting information to electronic device 600. Input unit 606 can receive input digital or character information and generate key signal input related to user settings and / or function control of electronic device, and can include, but is not limited to, a mouse, keyboard, touch screen, trackpad, trackball, joystick, microphone and / or remote control. Output unit can be any type of device capable of presenting information, and can include, but is not limited to, display unit 605, speaker 606 and other output units 609. Other output units 609 can include, but are not limited to, video / audio output terminals, vibrators and / or printers. Communication unit 608 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and can include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth devices, 802.6 devices, WiFi devices, WiMax devices, cellular communication devices and / or the like.

[0093] The application program 602b in working register 602 can be loaded to execute the various methods and processes described above, for example... Figure 1 Steps S102-S110 are described above. For example, in some embodiments, the method 100 described above may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 607. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via storage unit 607 and / or communication unit 608. When the computer program is loaded and executed by processor 601, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, processor 601 may be configured to perform method 100 by any other suitable means (e.g., by means of firmware).

[0094] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0095] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0096] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0097] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0098] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0099] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0100] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0101] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A video processing method, comprising: Determine the first and second consecutive frames in the video; The motion intensity between the first frame and the second frame is determined based on the motion information between the first frame and the second frame; Based on the motion intensity, an inter-frame optical flow map is determined between the first frame and the second frame, wherein the inter-frame optical flow map is obtained by processing the first frame and the second frame using an optical flow estimation model, and wherein, in response to determining that the motion intensity is greater than a threshold, the size of the first frame and the second frame is reduced to be used as input to the optical flow estimation model; as well as A target composite frame is determined based on the first frame, the second frame, and the inter-frame optical flow map, wherein the target composite frame is obtained using an optical flow-based image fusion model, and wherein the target composite frame is used to interpolate the video in a uniform interpolation manner.

2. The video processing method of claim 1, wherein, Determining the inter-frame optical flow map between the first frame and the second frame based on the motion intensity includes: The scaling factor for the inter-frame optical flow map is determined based on the motion intensity; The first frame and the second frame are scaled based on the scaling factor to obtain the scaled first frame and the scaled second frame. Optical flow estimation is performed on the scaled first frame and the scaled second frame to obtain a first optical flow between a first timestamp corresponding to the first frame and the target interpolation time, and a second optical flow between a second timestamp corresponding to the second frame and the target interpolation time, wherein the dimensions of the first optical flow and the second optical flow are the same as the dimensions of the scaled first frame and the scaled second frame; and The inter-frame optical flow map is determined based on the first optical flow and the second optical flow.

3. The video processing method of claim 2, wherein, Determining the target composite frame based on the first optical flow and the second optical flow includes: The first optical flow and the second optical flow are inversely scaled based on the scaling factor to obtain the inversely scaled first optical flow and the inversely scaled second optical flow, wherein the size of the inversely scaled first optical flow and the inversely scaled second optical flow is the same as the size of the first frame and the second frame; The target composite frame is determined based on the inversely scaled first optical flow, the inversely scaled second optical flow, the first frame, and the second frame.

4. The video processing method of claim 3, wherein, Determining the target composite frame based on the inversely scaled first optical flow, the inversely scaled second optical flow, the first frame, and the second frame includes: Based on the first optical flow after inverse scaling, a first sampling result is obtained by sampling from the first frame; Based on the inversely scaled second optical flow, a second sampling result is obtained by sampling from the second frame; and The first sampling result and the second sampling result are fused to obtain the target composite frame for the target interpolation time.

5. The video processing method as described in claim 2, wherein, Determining the scaling factor for the inter-frame optical flow map based on the motion intensity includes: The scaling factor corresponding to the motion intensity is determined according to a predetermined mapping relationship.

6. The video processing method according to any one of claims 1-5, wherein the motion information includes motion vectors or inter-frame optical flow between the first frame and the second frame.

7. The video processing method of claim 6, wherein determining the motion intensity between the first frame and the second frame based on motion information between the first frame and the second frame comprises: Based on the motion information, the maximum motion index between the first frame and the second frame is determined as a global motion parameter, wherein the global motion parameter indicates the motion intensity of the first frame and the second frame.

8. The video processing method as described in claim 7, wherein when the motion information is the inter-frame optical flow between the first frame and the second frame, the maximum motion index is the maximum optical flow between the first timestamp corresponding to the first frame and the target interpolation time or the maximum optical flow between the target interpolation time and the second timestamp corresponding to the second frame.

9. A video processing apparatus, comprising: The video frame determination unit is configured to determine the first and second consecutive frames in the video. The motion intensity determination unit is configured to determine the motion intensity between the first frame and the second frame based on motion information between the first frame and the second frame; An optical flow estimation unit is configured to determine an inter-frame optical flow map between the first frame and the second frame based on the motion intensity, wherein the inter-frame optical flow map is obtained by processing the first frame and the second frame through an optical flow estimation model, and wherein, in response to determining that the motion intensity is greater than a threshold, the size of the first frame and the second frame is reduced for input to the optical flow estimation model; The compositing unit is configured to determine a target compositing frame based on the first frame, the second frame, and the inter-frame optical flow map, wherein the target compositing frame is obtained using an optical flow-based image fusion model, and wherein the target compositing frame is used to interpolate the video in a uniform interpolation manner.

10. An electronic device, comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores a computer program that, when executed by the at least one processor, implements the method according to any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1-8.

12. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1-8.

Images