Video processing method and apparatus, device, and medium

Abstract

Embodiments of the present disclosure relate to a video processing method, device, equipment and medium, wherein the method comprises: determining a first optical flow of a first image block in a first video frame moving to a second video frame, and a second optical flow of a second image block in the second video frame moving to the first video frame, wherein the first video frame and the second video frame are adjacent video frames, and the first image block and the second image block are image regions comprising a plurality of pixel points; and synthesizing an intermediate video frame according to the first video frame, the second video frame, the first optical flow and the second optical flow, wherein the intermediate video frame is an estimated video frame to be inserted between the first video frame and the second video frame. It can be seen that the embodiments of the present disclosure reduce the calculation amount of the estimated video frame, so that the video frame rate can be improved in application scenarios with limited calculation amount such as mobile devices.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a video processing method, apparatus, device, and medium. Background Technology

[0002] Frame rate boosting technology can perform motion estimation between two video frames, and then generate an intermediate frame between the two video frames based on the motion estimation. Frame rate boosting technology can improve the smoothness of the picture and optimize the user's viewing experience.

[0003] In related technologies, intermediate frames can be generated based on pixel matching or deep learning models to improve the frame rate. However, the above technical solutions all generate a large amount of computation, so they are not suitable for implementation on devices with limited computing power, such as mobile devices. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides a video processing method, apparatus, device and medium.

[0005] In a first aspect, embodiments of this disclosure provide a video processing method, the method comprising:

[0006] Determine the first optical flow of a first image block moving from a first video frame to a second video frame, and the second optical flow of a second image block moving from a second video frame to the first video frame, wherein the first video frame and the second video frame are adjacent video frames, and the first image block and the second image block are image regions including multiple pixels.

[0007] An intermediate video frame is synthesized based on the first video frame, the second video frame, the first optical flow, and the second optical flow, wherein the intermediate video frame is an estimated video frame to be inserted between the first video frame and the second video frame.

[0008] In one optional implementation, determining the first optical flow of a first image block moving from a first video frame to a second video frame, and the second optical flow of a second image block moving from the second video frame to the first video frame, includes:

[0009] The first video frame is scaled to obtain a corresponding first image set, and the second video frame is scaled to obtain a corresponding second image set, wherein the first image set and the second image set include: multiple image layers with different resolutions;

[0010] Starting from the lowest resolution image layer in the first image set, calculate the initial optical flow of the pre-divided image blocks in the current layer image, and calculate the initial optical flow of the pre-divided image blocks in the next resolution image based on the initial optical flow of the image blocks in the current layer image, until the initial optical flow of the pre-divided image blocks in the highest resolution image layer is calculated, and determine it as the first optical flow of the first image block moving to the second video frame;

[0011] Starting from the lowest resolution image layer in the second image set, the initial optical flow of the pre-divided image blocks in the current layer image is calculated, and the initial optical flow of the pre-divided image blocks in the next resolution image is calculated based on the initial optical flow of the image blocks in the current layer image, until the initial optical flow of the pre-divided image blocks in the highest resolution image layer is calculated, and determined as the second optical flow of the second image block moving to the first video frame.

[0012] In one optional implementation, calculating the initial optical flow of pre-divided image patches in the current layer image includes:

[0013] Obtain the first-direction gradient value and the second-direction gradient value of each pixel in the image block of the current layer image;

[0014] Based on the first and second gradient values ​​of each pixel, determine the first pixel matrix, the second pixel matrix, and the third pixel matrix corresponding to the image block in the current layer image;

[0015] The first pixel matrix, the second pixel matrix, and the third pixel matrix are processed according to a preset algorithm to obtain the initial optical flow corresponding to the image block in the current layer image.

[0016] In one optional implementation, it further includes:

[0017] Anomaly detection is performed on the first optical flow of the first image block moving to the second video frame, and the second image block corresponding to the first image block moving to the second video frame is obtained based on the first optical flow of the first image block to be detected.

[0018] Calculate the first offset vector between the first optical flow of the current first image block to be detected and the second optical flow of the corresponding second image block in the second video frame, and compare the first offset vector with a preset first threshold.

[0019] If the first offset vector is greater than the first threshold, then compare the vector length of the first optical flow of the first image block to be detected with the inverse vector length of the second optical flow of the corresponding second image block in the second video frame;

[0020] If the inverse vector length of the second optical flow is less than the vector length of the first optical flow, then the first optical flow of the first image block to be detected is adjusted to the inverse vector of the second optical flow of the corresponding second image block in the second video frame.

[0021] In one optional implementation, it further includes:

[0022] Anomaly detection is performed on the second optical flow of the second image block moving to the first video frame, and the first image block corresponding to the second image block moving to the first video frame is obtained based on the second optical flow of the second image block to be detected.

[0023] Calculate the second offset vector between the second optical flow of the currently detected second image block and the first optical flow of the corresponding first image block in the first video frame, and compare the second offset vector with a preset second threshold.

[0024] If the second offset vector is greater than the second threshold, then compare the vector length of the second optical flow of the currently detected second image block with the inverse vector length of the first optical flow of the corresponding first image block in the first video frame;

[0025] If the inverse vector length of the first optical flow is less than the vector length of the second optical flow, then the second optical flow of the currently detected second image block is adjusted to the inverse vector of the first optical flow of the corresponding first image block in the first video frame.

[0026] In one optional implementation, it further includes:

[0027] Anomaly detection is performed on the first image block corresponding to the row boundary or column boundary in the first video frame, and the vector length corresponding to the first optical flow of the first image block to be detected at the row boundary or column boundary is obtained.

[0028] The vector length corresponding to the first optical flow of the first image block of the current row or column boundary to be detected is compared with a preset threshold value;

[0029] If the number of vector lengths less than the preset threshold is greater than the preset third threshold, then the first optical flow of the first image block of the current row boundary or column boundary to be detected is adjusted to the first optical flow of the first image block of the adjacent row or column of the current row boundary or column boundary to be detected.

[0030] And / or,

[0031] Anomaly detection is performed on the second image block corresponding to the row boundary or column boundary in the second video frame, and the vector length corresponding to the second optical flow of the second image block to be detected at the row boundary or column boundary is obtained.

[0032] The vector length corresponding to the second optical flow of the second image block at the current row or column boundary to be detected is compared with a preset threshold value;

[0033] If the number of vector lengths less than the preset threshold is greater than the preset third threshold, then the second optical flow of the second image block at the current row or column boundary to be detected is adjusted to the second optical flow of the second image block in the adjacent row or column of the current row or column boundary to be detected.

[0034] In one optional implementation, the step of synthesizing intermediate video frames based on the first video frame, the second video frame, the first optical flow, and the second optical flow includes:

[0035] Motion search and adjustment are performed on the first optical flow of the first image block moving to the second video frame to obtain the third optical flow of the first image block moving to the second video frame, and motion search and adjustment are performed on the second optical flow of the second image block moving to the first video frame to obtain the fourth optical flow of the second image block moving to the first video frame.

[0036] The intermediate video frame is synthesized based on the first video frame, the second video frame, the third optical flow of the first image block moving to the second video frame, and the fourth optical flow of the second image block moving to the first video frame.

[0037] In one optional implementation, the step of performing motion search adjustment on the first optical flow of the first image block moving to the second video frame, and obtaining the third optical flow of the first image block moving to the second video frame, includes:

[0038] Motion search is performed on the first image block to determine whether the first image block to be processed is located on the boundary of the first video frame. If the first image block to be processed is located on the boundary, no adjustment is made and the first optical flow of the first image block to be processed is used as the third optical flow to move to the second video frame.

[0039] If the first image block to be processed is not located at the boundary, then a first candidate vector array is established based on the first optical flow of the first image block to be processed, and the first candidate median of the first candidate vector array is determined.

[0040] Motion search is performed on the first image block within a first search vector range associated with the first candidate median, and a first target vector is determined within the first search vector range. The difference between the sum of all pixels of the image block in the second video frame corresponding to the first target vector and the sum of all pixels of the first image block to be processed is less than the difference between the sum of all pixels of the image block in the second video frame corresponding to other vectors within the first search vector range and the sum of all pixels of the first image block to be processed.

[0041] The first optical flow of the first image block to be processed is adjusted to the first target vector, which is then used as the third optical flow for the first image block to be processed to move to the second video frame.

[0042] In one optional implementation, the step of performing motion search adjustment on the second optical flow of the second image block moving to the first video frame to obtain the fourth optical flow of the second image block moving to the first video frame includes:

[0043] Motion search is performed on the second image block to determine whether the current second image block to be processed is located at the boundary of the second video frame. If the current second image block to be processed is located at the boundary, no adjustment is made and the second optical flow of the current second image block to be processed is used as the fourth optical flow to move to the first video frame.

[0044] If the current second image block to be processed is not located at the boundary, then a second candidate vector array is established based on the second optical flow of the current second image block to be processed, and the second candidate median of the second candidate vector array is determined.

[0045] Motion search is performed on the second image block within the second search vector range associated with the second candidate median, and a second target vector is determined within the second search vector range, wherein the difference between the sum of all pixels of the image block in the first video frame corresponding to the second target vector and the sum of all pixels of the currently to-be-processed second image block is less than the difference between the sum of all pixels of the image block in the first video frame corresponding to other vectors within the second search vector range and the sum of all pixels of the currently to-be-processed second image block.

[0046] The second optical flow of the currently unprocessed second image block is adjusted to the second target vector, which serves as the fourth optical flow for the currently unprocessed second image block to move to the first video frame.

[0047] In an optional implementation, the step of synthesizing an intermediate video frame based on the first video frame, the second video frame, the third optical flow of the first image block moving to the second video frame, and the fourth optical flow of the second image block moving to the first video frame includes:

[0048] Based on the third optical flow of the first image block moving to the second video frame and the insertion time of the intermediate video frame, determine the coordinates of the first center point on the intermediate video frame corresponding to the first image block;

[0049] Based on the coordinates of each first center point, a corresponding first sampling block is obtained by sampling on the first video frame, and a corresponding second sampling block is obtained by sampling on the second video frame;

[0050] The pixels of the first sampling block and the pixels of the second sampling block obtained according to the coordinates of each first center point are accumulated into the intermediate video frame;

[0051] Based on the fourth optical flow of the second image block moving to the first video frame and the insertion time of the intermediate video frame, the coordinates of the second center point corresponding to the second image block on the intermediate video frame are determined;

[0052] Based on the coordinates of each second center point, a corresponding third sampling block is obtained by sampling on the first video frame, and a corresponding fourth sampling block is obtained by sampling on the second video frame;

[0053] The pixels of the third sampling block and the pixels of the fourth sampling block, obtained according to the coordinates of each second center point, are accumulated into the intermediate video frame.

[0054] In one optional implementation, it further includes:

[0055] According to a preset bilinear kernel weight, the pixels of the first sampling block and the pixels of the second sampling block are accumulated into the intermediate video frame, and the pixels of the third sampling block and the pixels of the fourth sampling block are accumulated into the intermediate video frame. Secondly, embodiments of this disclosure provide a video processing apparatus, the apparatus comprising:

[0056] The determining module is used to determine the first optical flow of a first image block moving from a first video frame to a second video frame, and the second optical flow of a second image block moving from a second video frame to the first video frame, wherein the first video frame and the second video frame are adjacent video frames, and the first image block and the second image block are image regions including multiple pixels.

[0057] The compositing module is configured to compose an intermediate video frame based on the first video frame, the second video frame, the first optical flow, and the second optical flow, wherein the intermediate video frame is an estimated video frame to be inserted between the first video frame and the second video frame.

[0058] Thirdly, this disclosure provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to implement the above-described method.

[0059] Fourthly, this disclosure provides an apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method described above.

[0060] Fifthly, this disclosure provides a computer program product comprising a computer program / instruction that, when executed by a processor, implements the method described above.

[0061] The technical solution provided in this disclosure has at least the following advantages compared with the prior art:

[0062] This disclosure provides a video processing method. First, a first optical flow is determined when a first image block in a first video frame moves to a second video frame, and a second optical flow is determined when a second image block in the second video frame moves to the first video frame. The first and second video frames are adjacent video frames, and the first and second image blocks are image regions comprising multiple pixels. Further, an intermediate video frame is synthesized based on the first video frame, the second video frame, the first optical flow, and the second optical flow. This intermediate video frame is an estimated video frame to be inserted between the first and second video frames. Therefore, this disclosure improves the robustness and accuracy of video processing in scenes with large motion scales and reduces the computational cost of estimating video frames, enabling video frame rate enhancement in application scenarios with limited computational resources, such as mobile devices. Attached Figure Description

[0063] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0064] Figure 1 A flowchart illustrating a video processing method provided in an embodiment of this disclosure;

[0065] Figure 2 A flowchart illustrating another video processing method provided in this embodiment of the present disclosure;

[0066] Figure 3 A flowchart illustrating yet another video processing method provided in this disclosure embodiment;

[0067] Figure 4 A schematic diagram of an image pyramid provided for an embodiment of this disclosure;

[0068] Figure 5 A schematic diagram of an image block provided in an embodiment of this disclosure;

[0069] Figure 6 A schematic diagram illustrating the calculation of a first pixel matrix provided in a disclosed embodiment;

[0070] Figure 7 A schematic diagram of a loss value calculation method provided in this embodiment of the present disclosure;

[0071] Figure 8 A schematic diagram illustrating a loss value calculation method provided in an embodiment of this disclosure;

[0072] Figure 9 A schematic diagram of an intermediate video frame provided in an embodiment of this disclosure;

[0073] Figure 10 This is a schematic diagram of image block overlay provided in an embodiment of the present disclosure;

[0074] Figure 11 A flowchart illustrating yet another video processing method provided in this disclosure embodiment;

[0075] Figure 12 A flowchart illustrating yet another video processing method provided in this disclosure embodiment;

[0076] Figure 13 A flowchart illustrating yet another video processing method provided in this disclosure embodiment;

[0077] Figure 14 A schematic diagram illustrating the calculation of a second offset vector provided in an embodiment of this disclosure;

[0078] Figure 15 A flowchart illustrating yet another video processing method provided in this disclosure embodiment;

[0079] Figure 16 This is a schematic diagram of the structure of a video processing apparatus provided in an embodiment of the present disclosure;

[0080] Figure 17 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0081] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0082] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0083] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0084] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0085] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0086] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0087] To address the aforementioned problems, this disclosure provides a video processing method, which will be described below with reference to specific embodiments.

[0088] Figure 1 This is a flowchart illustrating a video processing method provided in an embodiment of the present disclosure. The method can be executed by a video processing device, which can be implemented using software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method includes:

[0089] Step 101: Determine the first optical flow of the first image block in the first video frame moving to the second video frame, and the second optical flow of the second image block in the second video frame moving to the first video frame, wherein the first video frame and the second video frame are adjacent video frames, and the first image block and the second image block are image regions including multiple pixels.

[0090] In this embodiment, to improve the video frame rate, estimated video frames need to be inserted between the first and second video frames, which are adjacent video frames. First, it is necessary to determine the bidirectional optical flow between the first and second video frames, specifically including: determining the first optical flow of a first image block moving from the first video frame to the second video frame, and the second optical flow of a second image block moving from the second video frame to the first video frame.

[0091] In this embodiment, the first video frame is divided into multiple first image blocks, each of which is an image region comprising multiple pixels. In this embodiment, the first video frame can be divided according to division parameters, which can be selected based on the application scenario. These division parameters include, but are not limited to, the side length of the first image block and / or the number of pixels between adjacent first image blocks. The first image blocks obtained by dividing the first video frame may or may not have overlapping pixels; this embodiment does not impose any restrictions.

[0092] Furthermore, the first optical flow corresponding to the first image block is determined based on the first image block. Understandably, the first optical flow reflects the motion estimate of the first image block moving from the first video frame to the second video frame. There are several methods available for calculating the first optical flow, which can be selected according to the application scenario, such as the pyramid Lucas-Kanade optical flow method.

[0093] In this embodiment, the second video frame is divided into multiple second image blocks, each of which is an image region comprising multiple pixels. In this embodiment, the second video frame can be divided according to division parameters, which can be selected based on the application scenario. These division parameters include, but are not limited to, the side length of the second image block and / or the number of pixels between adjacent second image blocks. The second image blocks obtained by dividing the second video frame may or may not have overlapping pixels; this embodiment does not impose any restrictions.

[0094] Furthermore, the second optical flow corresponding to the second image patch is determined based on the second image patch. Understandably, the second optical flow reflects the motion estimate of the second image patch moving from the second video frame to the first video frame. Several methods are available for calculating the second optical flow, which can be selected based on the application scenario, such as the pyramid Lucas-Kanade optical flow method.

[0095] Step 102: Synthesize an intermediate video frame based on the first video frame, the second video frame, the first optical flow, and the second optical flow, wherein the intermediate video frame is an estimated video frame to be inserted between the first video frame and the second video frame.

[0096] In this embodiment, the estimated video frame between the first video frame and the second video frame can both seamlessly connect to the first video frame and smoothly transition to the second video frame. In this embodiment, sampling can be performed on both the first and second video frames based on the first optical flow, and the sampled image blocks are accumulated onto the intermediate video frame according to the coordinates corresponding to the first optical flow. Furthermore, sampling can be performed on both the first and second video frames based on the second optical flow, and the sampled image blocks are accumulated onto the intermediate video frame according to the coordinates corresponding to the second optical flow. This intermediate video frame is then used as the estimated video frame inserted between the first and second video frames.

[0097] The video processing method provided in this disclosure determines a first optical flow from a first image block in a first video frame to a second video frame, and a second optical flow from a second image block in a second video frame to the first video frame. The first and second video frames are adjacent video frames, and the first and second image blocks are image regions comprising multiple pixels. An intermediate video frame is synthesized based on the first video frame, the second video frame, the first optical flow, and the second optical flow. This intermediate video frame is an estimated video frame to be inserted between the first and second video frames. Therefore, this disclosure improves the robustness and accuracy of video processing in scenes with large motion scales and reduces the computational cost of estimating video frames, enabling video frame rate enhancement in application scenarios with limited computational resources, such as mobile devices.

[0098] Figure 2 This is a flowchart illustrating another video processing method provided in this disclosure. In this method, motion search adjustments can be performed on the first optical flow and the second optical flow based on the above embodiments to achieve fine-tuning of the optical flow, such as... Figure 2 As shown, this includes the following steps:

[0099] Step 201: Determine the first optical flow of the first image block in the first video frame moving to the second video frame, and the second optical flow of the second image block in the second video frame moving to the first video frame, wherein the first video frame and the second video frame are adjacent video frames, and the first image block and the second image block are image regions including multiple pixels.

[0100] Step 202: Perform motion search and adjustment on the first optical flow of the first image block moving to the second video frame to obtain the third optical flow of the first image block moving to the second video frame, and perform motion search and adjustment on the second optical flow of the second image block moving to the first video frame to obtain the fourth optical flow of the second image block moving to the first video frame.

[0101] Furthermore, after determining the first optical flow, to further improve its accuracy, the first optical flow can be fine-tuned. Motion search can be used to obtain a third optical flow corresponding to the first optical flow in its vicinity, and the accuracy of this third optical flow will be better than that of the first optical flow. There are various algorithms for motion search of the first optical flow, which can be selected according to the application scenario. This embodiment does not impose any limitations; for example, hexagonal search algorithm and rhombus search algorithm.

[0102] Similar to the method described above for obtaining a third optical flow through motion search adjustment of the first optical flow, in this embodiment, to further improve the accuracy of the second optical flow, it can be fine-tuned. Motion search can then be used to obtain a fourth optical flow corresponding to the second optical flow in its vicinity, and the accuracy of this fourth optical flow will be better than that of the second optical flow. Various algorithms can be used for motion search of the second optical flow, and the appropriate algorithm can be selected based on the application scenario. This embodiment does not impose any limitations; for example, hexagonal search algorithm and rhombus search algorithm can be used.

[0103] The third optical flow obtained by motion search adjustment of the first image block in the first video frame moving to the second video frame and the fourth optical flow obtained by the second image block in the second video frame moving to the first video frame can more accurately represent the motion of detailed areas such as dense textures.

[0104] Step 203: Synthesize an intermediate video frame based on the first video frame, the second video frame, the third optical flow of the first image block moving to the second video frame, and the fourth optical flow of the second image block moving to the first video frame.

[0105] In this embodiment, the estimated video frame between the first video frame and the second video frame can both seamlessly connect to the first video frame and smoothly transition to the second video frame. In this embodiment, sampling can be performed on the first and second video frames respectively based on a third optical flow, and the sampled image blocks are accumulated onto the intermediate video frame according to the coordinates corresponding to the third optical flow. Furthermore, sampling can be performed on the first and second video frames respectively based on a fourth optical flow, and the sampled image blocks are accumulated onto the intermediate video frame according to the coordinates corresponding to the fourth optical flow. This intermediate video frame is then used as the estimated video frame inserted between the first and second video frames.

[0106] The video processing method provided in this disclosure determines the first optical flow of a first image block moving from a first video frame to a second video frame, and the second optical flow of a second image block moving from a second video frame to the first video frame. The first and second video frames are adjacent video frames, and the first and second image blocks are image regions comprising multiple pixels. Motion search adjustment is performed on the first optical flow of the first image block moving to the second video frame to obtain a third optical flow. Similarly, motion search adjustment is performed on the second optical flow of the second image block moving to the first video frame to obtain a fourth optical flow. An intermediate video frame is synthesized based on the first video frame, the second video frame, the third optical flow of the first image block moving to the second video frame, and the fourth optical flow of the second image block moving to the first video frame. Therefore, this disclosure improves the robustness and accuracy of video processing in scenes with large motion scales and achieves fine-tuning of optical flow, thereby reducing the computational load of estimating video frames. This allows for video frame rate improvement in application scenarios with limited computational resources, such as mobile devices. Furthermore, the fine-tuning of optical flow can further improve the accuracy of optical flow in detailed regions such as dense textures.

[0107] Figure 3 A flowchart illustrating yet another video processing method provided in this disclosure embodiment is shown below. Figure 3 As shown, the method includes the following steps:

[0108] Step 301: Scaling the first video frame to obtain the corresponding first image set, and scaling the second video frame to obtain the corresponding second image set, wherein the first image set and the second image set include: multiple image layers with different resolutions.

[0109] In this embodiment, scaling processing can scale the first video frame to different resolution scales, thereby obtaining different resolution layers for the first video frame. Then, a first image set is established based on these different resolution layers. This first image set can be, for example,... Figure 4 The image pyramid shown has an image layer resolution that increases sequentially from top to bottom, consisting of the first set of images.

[0110] Similarly, scaling can be used to scale the second video frame to different resolution scales, thereby obtaining different resolution layers for the second video frame. Then, a second image set can be built based on these different resolution layers. This second image set can also be, for example... Figure 4 The image pyramid shown is composed of a second set of images, in which the resolution of the image layers increases sequentially from the top to the bottom.

[0111] Step 302: Starting from the lowest resolution image layer in the first image set, calculate the initial optical flow of the pre-divided image blocks in the current layer image, and calculate the initial optical flow of the pre-divided image blocks in the next resolution image based on the initial optical flow of the image blocks in the current layer image, until the initial optical flow of the pre-divided image blocks in the highest resolution image layer is calculated, and determine it as the first optical flow of the first image block moving to the second video frame.

[0112] In this embodiment, dividing the image layer according to the patch length (patch_size) and patch stride (patch_stride) yields corresponding image patches. The patch length represents the number of pixels along one side of the image patch, and the patch stride represents the number of pixels between adjacent image patches. These patch length and patch stride can be set according to the application scenario, etc., and this embodiment does not impose any limitations. One optional implementation method is... Figure 5 This is a schematic diagram of an image block provided in an embodiment of the present disclosure, such as... Figure 5 As shown, Figure 5 Each grid in the diagram represents a pixel. Figure 5 In the diagram, the nine bolded squares represent a single image block. This block has a side length of 3 pixels and a spacing of 2 pixels. Figure 5 The solid squares in the center represent the center pixels of each image block.

[0113] According to the image block division rule, the image layers in the first image set are divided into image blocks. Then, starting from the lowest resolution image layer in the first image set, the initial optical flow of the pre-divided image blocks in the current layer image is calculated. Based on the initial optical flow of the image blocks in the current layer image, the initial optical flow of the corresponding pre-divided image blocks in the next layer resolution image is calculated until the initial optical flow of the pre-divided image blocks in the highest resolution image layer is calculated. This initial optical flow is then determined as the first optical flow of the first image block moving to the second video frame.

[0114] To illustrate this more clearly, let's take the first image set as an example. Figure 4 Taking the image pyramid shown as an example, the initial optical flow of the pre-divided image blocks in the topmost image layer of the image pyramid is first calculated. Then, the initial optical flow of the image blocks in the next image layer of the image pyramid is calculated sequentially based on the initial optical flow of the image blocks in the current image layer, until the initial optical flow of the pre-divided image blocks in the bottommost image layer of the image pyramid is obtained. The initial optical flow of the pre-divided image blocks in the bottommost image layer is determined as the first optical flow of the first image block moving to the second video frame. In this embodiment, by calculating the initial optical flow of the pre-divided image blocks in the next resolution image based on the initial optical flow of the image blocks in the current layer image, the calculated second optical flow can more accurately represent motion of different amplitudes.

[0115] In an optional implementation, the step of calculating the initial optical flow of pre-divided image blocks in the current layer image includes:

[0116] Step a1: Obtain the first-direction gradient value and the second-direction gradient value of each pixel in the image patch of the current layer image.

[0117] In this embodiment, the first direction and the second direction are different from each other.

[0118] In one optional implementation, the first direction and the second direction are perpendicular to each other, the first direction is the x-direction and the second direction is the y-direction. Accordingly, the first direction gradient value dx and the second direction gradient value dy of each pixel of the image block in the current layer image are obtained.

[0119] Step a2: Determine the first pixel matrix, second pixel matrix, and third pixel matrix corresponding to the image patch in the current layer image based on the first and second gradient values ​​of each pixel.

[0120] In this embodiment of the disclosure, the first pixel matrix, the second pixel matrix, and the third pixel matrix are matrices determined based on the gradient values ​​in the first direction and / or the gradient values ​​in the second direction, and these matrices correspond to the center pixels of the image blocks.

[0121] In one optional implementation, the gradient value of the first direction of each pixel in an image patch can be squared and summed to obtain the element value corresponding to that image patch in the first pixel matrix. This operation is performed on each image patch in the current layer image to obtain the element value corresponding to each image patch in the first pixel matrix. The first pixel matrix is ​​then filled according to the positional relationship between the image patches to obtain the first pixel matrix. If the width of the current layer image is W pixels and the height is H pixels, and the current layer image is divided according to the patch side length `patch_size` and the patch interval `patch_stride`, then the first pixel matrix has W / patch_stride columns and H / patch_stride rows.

[0122] For example, such as Figure 6 As shown, Figure 6 This is a schematic diagram illustrating the calculation of a first pixel matrix provided in a disclosed embodiment. Figure 6 In the image on the left, each grid represents one pixel. Figure 6 The nine bolded squares in the middle border represent an example image block, which consists of nine pixels. The square of the first-direction gradient value of each pixel is calculated, ranging from q0 to q8. The squares of these nine first-direction gradient values ​​are then summed to obtain the element value p corresponding to this example image block in the first pixel matrix. Figure 6The right image in the image is composed of the center pixels of each image patch in the left image. The center pixel of this example image patch is located in the second row and second column of the right image. Therefore, the element value corresponding to this example image patch is also located in the second row and second column of the first pixel matrix. By calculating for each image patch in the current layer image, the corresponding first pixel matrix can be obtained. Figure 6 The width of the current layer image is 7 pixels and the height is 5 pixels. The calculated first pixel matrix has 4 columns and 3 rows.

[0123] Similarly, the second-direction gradient value of each pixel in the image block is squared and summed to obtain the element value of the image block in the second pixel matrix. The above operation is performed on each image block in the current layer image to obtain the element value of each image block in the second pixel matrix. The second pixel matrix is ​​then filled according to the positional relationship between the image blocks to obtain the second pixel matrix.

[0124] Multiply the first and second gradient values ​​of each pixel in the image block and sum them to obtain the element value of the corresponding image block in the third pixel matrix. Perform the above operation on each image block in the current layer image to obtain the element value of each image block in the third pixel matrix. Then fill the third pixel matrix according to the positional relationship between the image blocks to obtain the third pixel matrix.

[0125] Step a3: Process the first pixel matrix, the second pixel matrix, and the third pixel matrix according to the preset algorithm to obtain the initial optical flow corresponding to the image block in the current layer image.

[0126] In this embodiment, the preset algorithm can calculate the initial optical flow corresponding to the image block in the current layer image based on the first pixel matrix, the second pixel matrix, and the third pixel matrix. There are multiple preset algorithms, which can be selected according to the application scenario, etc., and this embodiment does not impose any restrictions.

[0127] In one optional implementation, the optical flow update value Δu can be calculated based on the first pixel matrix, the second pixel matrix, and the third pixel matrix. The optical flow value to be precisely calculated, u, is added to the optical flow update value Δu to update the optical flow value to be precisely calculated u. Assuming there is an image block centered at pixel p in the first video frame, taking this image block as an example, the optical flow update value Δu for this image block is:

[0128]

[0129] In the above formula, T represents the image block centered at pixel p in the first video frame, T(x) represents the value of pixel x in the image block, S represents the gradient of T, I1 represents the second video frame, and ∑ x S T[I1(x+u)-T(x)] represents the S-value of x pixels in the image patch. T [I1(x+u)-T(x)] is used for summation, where H is the Hessian matrix of the center pixel of the image patch in the current layer image. Specifically:

[0130]

[0131] in, The value corresponding to pixel p in the first pixel matrix. The value corresponding to pixel p in the second pixel matrix. This is the value corresponding to pixel p in the third pixel matrix.

[0132] It should be noted that when initially calculating the optical flow update value Δu, the optical flow value u to be precisely calculated can be set to 0, and the optical flow update value Δu can be calculated iteratively to update the optical flow value u to be precisely calculated. When the number of iterations meets the preset number of iterations, the new optical flow value u to be precisely calculated is determined as the initial light, and the calculation is performed on each image patch in the current layer image to obtain the initial optical flow corresponding to the image patch in the current layer image. The preset number of iterations can be set according to the application scenario, for example, 5 times.

[0133] Step 303: Starting from the lowest resolution image layer in the second image set, calculate the initial optical flow of the pre-divided image blocks in the current layer image, and calculate the initial optical flow of the pre-divided image blocks in the next resolution image based on the initial optical flow of the image blocks in the current layer image, until the initial optical flow of the pre-divided image blocks in the highest resolution image layer is calculated, and determine it as the second optical flow of the second image block moving to the first video frame.

[0134] Based on the same image block partitioning rules as the above steps, the image layers in the second image set are partitioned into image blocks. Then, starting from the lowest resolution image layer in the second image set, the initial optical flow of the pre-divided image blocks in the current layer image is calculated. Based on the initial optical flow of the image blocks in the current layer image, the initial optical flow of the corresponding pre-divided image blocks in the next layer resolution image is calculated until the initial optical flow of the pre-divided image blocks in the highest resolution image layer is calculated. This initial optical flow is then determined as the second optical flow of the second image block moving to the first video frame.

[0135] To illustrate this more clearly, let's take the second image set as an example. Figure 4Taking the image pyramid shown as an example, the initial optical flow of the pre-divided image blocks in the topmost image layer of the image pyramid is first calculated. Then, the initial optical flow of the image blocks in the next image layer of the image pyramid is calculated sequentially based on the initial optical flow of the image blocks in the current image layer, until the initial optical flow of the pre-divided image blocks in the bottommost image layer of the image pyramid is obtained. The initial optical flow of the pre-divided image blocks in the bottommost image layer is determined as the second optical flow of the second image block moving to the first video frame. In this embodiment, by calculating the initial optical flow of the pre-divided image blocks in the next resolution image based on the initial optical flow of the image blocks in the current layer image, motion of different amplitudes can be accurately represented.

[0136] In an optional implementation, the step of calculating the initial optical flow of pre-divided image blocks in the current layer image includes:

[0137] Step b1: Obtain the first-direction gradient value and the second-direction gradient value of each pixel in the image patch of the current layer image.

[0138] In this embodiment, the first direction and the second direction are different from each other.

[0139] In one optional implementation, the first direction and the second direction are perpendicular to each other, the first direction is the x-direction and the second direction is the y-direction. Accordingly, the first direction gradient value dx and the second direction gradient value dy of each pixel of the image block in the current layer image are obtained.

[0140] Step b2: Determine the first pixel matrix, second pixel matrix, and third pixel matrix corresponding to the image patch in the current layer image based on the first and second gradient values ​​of each pixel.

[0141] In this embodiment of the disclosure, the first pixel matrix, the second pixel matrix, and the third pixel matrix are matrices determined based on the gradient values ​​in the first direction and / or the gradient values ​​in the second direction, and these matrices correspond to the center pixels of the image blocks.

[0142] In one optional implementation, the gradient value of the first direction of each pixel in an image patch can be squared and summed to obtain the element value corresponding to that image patch in the first pixel matrix. This operation is performed on each image patch in the current layer image to obtain the element value corresponding to each image patch in the first pixel matrix. The first pixel matrix is ​​then filled according to the positional relationship between the image patches to obtain the first pixel matrix. If the width of the current layer image is W pixels and the height is H pixels, and the current layer image is divided according to the patch side length `patch_size` and the patch interval `patch_stride`, then the first pixel matrix has W / patch_stride columns and H / patch_stride rows.

[0143] For example, such as Figure 6 As shown, Figure 6 This is a schematic diagram illustrating the calculation of a first pixel matrix provided in a disclosed embodiment. Figure 6 In the image on the left, each grid represents one pixel. Figure 6 The nine bolded squares in the middle border represent an example image block, which consists of nine pixels. The square of the first-direction gradient value of each pixel is calculated, ranging from q0 to q8. The squares of these nine first-direction gradient values ​​are then summed to obtain the element value p corresponding to this example image block in the first pixel matrix. Figure 6 The right image in the image is composed of the center pixels of each image patch in the left image. The center pixel of this example image patch is located in the second row and second column of the right image. Therefore, the element value corresponding to this example image patch is also located in the second row and second column of the first pixel matrix. By calculating for each image patch in the current layer image, the corresponding first pixel matrix can be obtained. Figure 6 The width of the current layer image is 7 pixels and the height is 5 pixels. The calculated first pixel matrix has 4 columns and 3 rows.

[0144] Similarly, the second-direction gradient value of each pixel in the image block is squared and summed to obtain the element value of the image block in the second pixel matrix. The above operation is performed on each image block in the current layer image to obtain the element value of each image block in the second pixel matrix. The second pixel matrix is ​​then filled according to the positional relationship between the image blocks to obtain the second pixel matrix.

[0145] Multiply the first and second gradient values ​​of each pixel in the image block and sum them to obtain the element value of the corresponding image block in the third pixel matrix. Perform the above operation on each image block in the current layer image to obtain the element value of each image block in the third pixel matrix. Then fill the third pixel matrix according to the positional relationship between the image blocks to obtain the third pixel matrix.

[0146] Step b3: Process the first pixel matrix, the second pixel matrix, and the third pixel matrix according to the preset algorithm to obtain the initial optical flow corresponding to the image block in the current layer image.

[0147] In this embodiment, the preset algorithm can calculate the initial optical flow corresponding to the image block in the current layer image based on the first pixel matrix, the second pixel matrix, and the third pixel matrix. There are multiple preset algorithms, which can be selected according to the application scenario, etc., and this embodiment does not impose any restrictions.

[0148] In one optional implementation, the optical flow update value Δu can be calculated based on the first pixel matrix, the second pixel matrix, and the third pixel matrix. The optical flow value to be precisely calculated, u, is added to the optical flow update value Δu to update the optical flow value to be precisely calculated u. Assuming there is an image block centered at pixel p in the first video frame, taking this image block as an example, the optical flow update value Δu for this image block is:

[0149]

[0150] In the above formula, T represents the image block centered at pixel p in the second video frame, T(x) represents the value of pixel x in the image block, S represents the gradient of T, I0 represents the first video frame, and ∑ x S T [I0(x+u)-T(x)] represents the S-value of x pixels in the image patch. T [I0(x+u)-T(x)] is used for summation, where H is the Hessian matrix of the center pixel of the image patch in the current layer image. Specifically:

[0151]

[0152] in, The value corresponding to pixel p in the first pixel matrix. The value corresponding to pixel p in the second pixel matrix. This is the value corresponding to pixel p in the third pixel matrix.

[0153] It should be noted that when initially calculating the optical flow update value Δu, the optical flow value u to be precisely calculated can be set to 0, and the optical flow update value Δu can be calculated iteratively to update the optical flow value u to be precisely calculated. When the number of iterations meets the preset number of iterations, the new optical flow value u to be precisely calculated is determined as the initial light, and the calculation is performed on each image patch in the current layer image to obtain the initial optical flow corresponding to the image patch in the current layer image. The preset number of iterations can be set according to the application scenario, for example, 5 times.

[0154] The methods for obtaining the first and second optical flows provided in the above steps can be executed in parallel, thereby improving computational efficiency.

[0155] It should be noted that the size of the optical flow map composed of the first optical flow and the second optical flow obtained through the above steps is W / patch_stride*H / patch_stride, where W is the number of pixels in the width direction of the current layer image, H is the number of pixels in the height direction of the current layer image, and patch_stride is the image patch spacing. Optionally, the optical flow map can also be scaled to transform it into a denser optical flow map of size W*H.

[0156] Specifically, the densed optical flow map includes image patch center points and non-image patch center points. The optical flow of the image patch center points in the densed optical flow map can be determined based on the optical flow in the optical flow map of size W / patch_stride*H / patch_stride. The optical flow of the non-image patch center points in the densed optical flow map can be the average of the optical flow of multiple image patch center points that are adjacent to or share a vertex with the non-image patch center point.

[0157] Step 304: Perform motion search on the first image block to determine whether the first image block to be processed is located at the boundary of the first video frame. If the first image block to be processed is located at the boundary, no adjustment is made and the first optical flow of the first image block to be processed is used as the third optical flow to move to the second video frame.

[0158] In one optional implementation, if the boundary of the first image block coincides with the boundary of the current layer image, or if the boundary of the first image block exceeds the boundary of the current layer image, then the first image block to be processed can be considered to be located at the boundary of the first video frame.

[0159] Motion search is performed on the first image block to determine whether the first image block to be processed is located at the boundary of the first video frame. If the first image block to be processed is located at the boundary, no adjustment is made to the first image block to be processed, and the first optical flow of the first image block to be processed is used as the third optical flow to move to the second video frame.

[0160] Step 305: If the first image block to be processed is not located at the boundary, then establish a first candidate vector array based on the first optical flow of the first image block to be processed, and determine the first candidate median of the first candidate vector array.

[0161] If the first image patch to be processed is not located at the boundary, a first candidate vector array is established based on the first optical flow of the first image patch to be processed. The first candidate vector array includes multiple optical flows related to the first optical flow of the first image patch, and the first candidate median of the first candidate vector array is determined.

[0162] In one optional implementation, the image block adjacent to the top of the first image block to be processed is the first upper image block, the image block adjacent to the bottom is the first lower image block, the image block adjacent to the left is the first left image block, and the image block adjacent to the right is the first right image block. The first candidate vector array includes: the first optical flow of the first upper image block, the first optical flow of the first lower image block, the first optical flow of the first left image block, the first optical flow of the first right image block, and the zero optical flow (0, 0), and the median of the above five optical flows is taken as the first candidate median.

[0163] Step 306: Perform motion search on the first image block within the first search vector range associated with the first candidate median, and determine the first target vector within the first search vector range. The difference between the sum of all pixels of the image block in the second video frame corresponding to the first target vector and the sum of all pixels of the first image block to be processed is less than the difference between the sum of all pixels of the image block in the second video frame corresponding to other vectors within the first search vector range and the sum of all pixels of the first image block to be processed.

[0164] In this embodiment, the first search vector range can be multiple vectors obtained by fine-tuning the elements of the first candidate median in different ways.

[0165] In one optional implementation, assume the first candidate median vector is and The first search vector range includes u1, u2, u3, and u4, where: And calculate The loss value cost of u1, u2, u3, and u4 is determined. The vector with the smallest loss value among u1, u2, u3, and u4. min and u min Assign to the first candidate median vector Continue calculating the current first candidate median vector Given the corresponding u1, u2, u3, and u4, determine the vector u with the smallest loss value. min until According to this The calculated u min Until they are equal, that is when it is determined. This is the first target vector.

[0166] Figure 7 This is a schematic diagram of a loss value calculation method provided in an embodiment of the present disclosure, as shown below. Figure 7 As shown, Figure 7 In this context, I0 represents the first video frame and I1 represents the second video frame. For the sake of brevity, I0 can also be used to represent the first video frame and I1 to represent the second video frame in the following embodiments. Figure 7 The vector to be calculated is the vector indicated by the arrow in I0. The first image block corresponding to this vector in the first video frame is the solid-marked grid in I0. The image block corresponding to this vector in the second video frame is the solid-marked grid in I1. The sum of the errors between all pixels of image block B1 in the second video frame and all pixels of the first image block B0 to be processed is taken as the loss value cost, i.e., cost = Sum(abs(B0-B1)), where abs() means taking the absolute value and Sum() means summing.

[0167] Step 307: Adjust the first optical flow of the first image block to be processed to the first target vector, and use it as the third optical flow of the first image block to be processed to move to the second video frame.

[0168] Then, the first optical flow of the first image block to be processed is adjusted to the first target vector confirmed by the above calculation, and the first target vector is used as the third optical flow of the first image block to be processed to move to the second video frame.

[0169] Step 308: Perform motion search on the second image block to determine whether the current second image block to be processed is located at the boundary of the second video frame. If the current second image block to be processed is located at the boundary, no adjustment is made and the second optical flow of the current second image block to be processed is used as the fourth optical flow to move to the first video frame.

[0170] In one optional implementation, if the boundary of the second image block coincides with the boundary of the current layer image, or if the boundary of the second image block exceeds the boundary of the current layer image, then the second image block to be processed can be considered to be located at the boundary of the second video frame.

[0171] Motion search is performed on the second image block to determine whether the current second image block to be processed is located at the boundary of the second video frame. If the current second image block to be processed is located at the boundary, no adjustment is made to the current second image block to be processed, and the second optical flow of the current second image block to be processed is used as the fourth optical flow to move to the first video frame.

[0172] Step 309: If the second image block to be processed is not located at the boundary, then establish a second candidate vector array based on the second optical flow of the second image block to be processed, and determine the second candidate median of the second candidate vector array.

[0173] If the second image patch to be processed is not located at the boundary, a second candidate vector array is established based on the second optical flow of the second image patch to be processed. The second candidate vector array includes multiple optical flows related to the second optical flow of the second image patch, and the second candidate median of the second candidate vector array is determined.

[0174] In one optional implementation, the image block adjacent to the top of the second image block to be processed is the second upper image block, the image block adjacent to the bottom is the second lower image block, the image block adjacent to the left is the second left image block, and the image block adjacent to the right is the second right image block. The second candidate vector array includes the second optical flow of the second upper image block, the second optical flow of the second lower image block, the second optical flow of the second left image block, the second optical flow of the second right image block, and the zero optical flow (0, 0). The median of the above five optical flows is taken as the second candidate median.

[0175] Step 310: Perform motion search on the second image block within the second search vector range associated with the second candidate median, and determine the second target vector within the second search vector range. The difference between the sum of all pixels of the image block in the first video frame corresponding to the second target vector and the sum of all pixels of the second image block to be processed is less than the difference between the sum of all pixels of the image block in the first video frame corresponding to other vectors within the second search vector range and the sum of all pixels of the second image block to be processed.

[0176] In this embodiment, the range of the second search vector can be multiple vectors obtained by fine-tuning the elements of the second candidate median in different ways.

[0177] In one optional implementation, assume the second candidate median vector is and The second search vector range includes u1′, u2′, u3′, and u4′, where:

[0178]

[0179] Further calculation The loss value cost′ of u1′, u2′, u3′, and u4′ is determined. The vector u with the smallest loss value among u1′, u2′, u3′, and u4′. min ′, and u min Assign the value to the second candidate median vector Continue calculating the current second candidate median vector Given the corresponding u1′, u2′, u3′, and u4′, determine the vector u with the smallest loss value. min ′, until According to this The calculated u min Until they are equal, determine the value at this point. This is the second target vector.

[0180] Figure 8 This is a schematic diagram of a loss value calculation method provided in an embodiment of the present disclosure, as shown below. Figure 8 As shown, Figure 8 The vector of loss values ​​to be calculated is I. ！The vector indicated by the middle arrow represents the second image block in the second video frame, which is the solid-marked grid in I1. The image block in the first video frame, which is the solid-marked grid in I0, represents the loss value vector to be calculated. The sum of the errors between all pixels of image block B0 in the first video frame and all pixels of the second image block B1 to be processed is taken as the loss value cost′, that is, cost′=Sum(abs(B1-B0)), where abs() means taking the absolute value and Sum() means summing.

[0181] Step 311: Adjust the second optical flow of the current second image block to be processed to the second target vector, and use it as the fourth optical flow of the current second image block to be processed to move to the first video frame.

[0182] Then, the second optical flow of the second image block to be processed is adjusted to the second target vector confirmed by the above calculation, and the second target vector is used as the fourth optical flow of the second image block to be processed to the first video frame.

[0183] Step 312: Based on the first video frame, the second video frame, the third optical flow of the first image block moving to the second video frame, and the fourth optical flow of the second image block moving to the first video frame, an intermediate video frame is synthesized, wherein the intermediate video frame is an estimated video frame to be inserted between the first video frame and the second video frame.

[0184] In one optional embodiment, the method for synthesizing an intermediate video frame based on a first video frame, a third optical flow from a first image block moving to a second video frame, and a fourth optical flow from a second video frame and a second image block moving to a first video frame includes the following steps:

[0185] Step c1: Based on the third optical flow of the first image block moving to the second video frame and the insertion time of the intermediate video frame, determine the coordinates of the first center point on the intermediate video frame corresponding to the first image block.

[0186] In this embodiment, the insertion time of the intermediate video frame can be set according to the application scenario. For example, if the time interval between the first video frame and the second video frame is set to a unit interval time of 1, the insertion time of the intermediate video frame can be a value between 0 and 1.

[0187] In one optional implementation, if the center point coordinates of the current first image block in the first video frame are (x0, y0), and the third optical flow is (mv x ,mv y If the insertion time is t, then in the coordinates of the first center point (center_x, center_y), center_x = int(x0 + t * mv) x ), center_y = int(y0 + t * mvy ), where int() represents taking an integer, and the value of t can be set according to the application scenario, for example: the value of t is 0.3.

[0188] Step c2: Based on the coordinates of each first center point, sample the first video frame to obtain the corresponding first sampling block, and sample the second video frame to obtain the corresponding second sampling block.

[0189] In this embodiment, the abscissa of the first video frame sampling coordinates on the first video frame can be determined based on the abscissa of the first center coordinate point, and the ordinate of the first video frame sampling coordinates on the first video frame can be determined based on the ordinate of the first center coordinate point, thereby obtaining a first sampling block by sampling on the first video frame according to the first video frame sampling coordinates; and the abscissa of the second video frame sampling coordinates on the second video frame can be determined based on the abscissa of the first center coordinate point, and the ordinate of the second video frame sampling coordinates on the second video frame can be determined based on the ordinate of the first center coordinate point, thereby obtaining a second sampling block by sampling on the second video frame according to the second video frame sampling coordinates.

[0190] Continuing with the first center point coordinates as (int(x0+t*mv) x ), int(y0+t*mv y Taking the example of ), the sampling coordinates of the first video frame determined based on the first center coordinates can be:

[0191] (int(x0+t*mv x )-t*mv x int(y0+t*mv) y )-t*mv y ).

[0192] On the first video frame, the first sampling block is obtained with the sampling coordinates of the first video frame as the center point.

[0193] Accordingly, the sampling coordinates of the second video frame determined based on the first center coordinates can be:

[0194] (int(x0+t*mv x )-(1-t)*mv x int(y0+t*mv) y )-(1-t)*mv y ).

[0195] On the second video frame, the second sampling block is obtained with the sampling coordinates of the second video frame as the center point.

[0196] In one optional implementation, the size of the first sampling block and the second sampling block can both be 32 pixels * 32 pixels.

[0197] Step c3: Based on the coordinates of each first center point, the pixels of the corresponding first sampling block and the pixels of the second sampling block are accumulated into the intermediate video frame.

[0198] After determining the first sampling block and the second sampling block, the first sampling block and the second sampling block are accumulated into the intermediate video frame according to the corresponding first center point coordinates.

[0199] To explain more clearly, such as Figure 9 As shown, Figure 9 This is a schematic diagram of an intermediate video frame provided in an embodiment of this disclosure. Figure 9 In this diagram, I0 represents the first video frame, I1 the second video frame, and It the intermediate video frame. The coordinates of the first center point in It are (center_x, center_y). The grid in I0 represents the first image block, which is 16 pixels * 16 pixels in size. The third optical flow of the first image block is traversed, and its size is expanded to 32 pixels * 32 pixels for motion compensation. For example, the shaded area centered at p in I0 represents the first sampling block with a size of 32 pixels * 32 pixels, and the shaded area centered at q in I1 represents the second sampling block with a size of 32 pixels * 32 pixels. Both the first and second sampling blocks are represented by the intermediate video frame I0. t The coordinates (center_x, center_y) are used as the center point to accumulate data to the intermediate video frame I. t superior.

[0200] Step c4: Based on the fourth optical flow of the second image block moving to the first video frame and the insertion time of the intermediate video frame, determine the coordinates of the second center point corresponding to the second image block on the intermediate video frame.

[0201] In one optional implementation, if the center point coordinates of the current second image block in the second video frame are (x0′, y0′), and the fourth optical flow is (mv x ′,mv y Given that the insertion time is t, the coordinates of the second center point (center_x′, center_y′) are:

[0202] center_x′=int(x0′+(1-t)*mv x ′),

[0203] center_y′=int(y0′+(1-t)*mv y ′).

[0204] Here, int() represents taking an integer, and the value of t can be set according to the application scenario. For example, the value of t is 0.3.

[0205] Step c5: Based on the coordinates of each second center point, sample the corresponding third sampling block on the first video frame, and sample the corresponding fourth sampling block on the second video frame.

[0206] In this embodiment, the abscissa of the first video frame sampling coordinates on the first video frame can be determined based on the abscissa of the second center coordinate point, and the ordinate of the first video frame sampling coordinates on the first video frame can be determined based on the ordinate of the second center coordinate point, thereby obtaining a third sampling block by sampling on the first video frame according to the first video frame sampling coordinates; and the abscissa of the second video frame sampling coordinates on the second video frame can be determined based on the abscissa of the second center coordinate point, and the ordinate of the second video frame sampling coordinates on the second video frame can be determined based on the ordinate of the second center coordinate point, thereby obtaining a fourth sampling block by sampling on the second video frame according to the second video frame sampling coordinates.

[0207] Continuing with the coordinates of the second center point (center_x′, center_y′):

[0208] (int(x0′+(1-t)*mv x ′), int(y0′+(1-t)*mv y Taking ')) as an example, the sampling coordinates of the first video frame determined based on the second center coordinates can be:

[0209] (center x ′-(1-t)*mv x ′, center y ′-(1-t)*mv y ′).

[0210] On the first video frame, the third sampling block is obtained with the sampling coordinates of the first video frame as the center point.

[0211] Accordingly, the sampling coordinates of the second video frame determined based on the second center coordinates can be:

[0212] (center x ′-t*mv x center y ′-t*mv y ).

[0213] On the second video frame, the fourth sampling block is obtained with the sampling coordinates of the second video frame as the center point.

[0214] In one optional implementation, the size of the third sampling block and the fourth sampling block can both be 32 pixels * 32 pixels.

[0215] Step c6: Based on the coordinates of each second center point, the pixels of the corresponding third sampling block and the pixels of the fourth sampling block are accumulated into the intermediate video frame.

[0216] After determining the third and fourth sampling blocks, the third and fourth sampling blocks are accumulated into the intermediate video frame according to the corresponding second center point coordinates.

[0217] Step c7: The pixels of the first sampling block and the pixels of the second sampling block are accumulated into the intermediate video frame according to the preset bilinear kernel weights, and the pixels of the third sampling block and the pixels of the fourth sampling block are accumulated into the intermediate video frame.

[0218] In the above steps, during the accumulation process to the intermediate video frame, there may be overlapping of image blocks. In this embodiment, the pixels of the first sampling block, the pixels of the second sampling block, the pixels of the third sampling block, and the pixels of the fourth sampling block can be accumulated to the intermediate video frame according to the preset bilinear kernel weight, thereby realizing the processing of overlapping image blocks.

[0219] For example, Figure 10 This is a schematic diagram of image block overlay provided in an embodiment of the present disclosure, such as... Figure 10 As shown, the first image block centered at p1 and the first image block centered at p2 in the first video frame I0 are superimposed on the intermediate video frame I. t There was an overlap, and the overlapping part was I. t The dark gray portion in the graph is weighted using bilinear kernel weights.

[0220] The size and specific parameters of the bilinear kernel weights in the above embodiments can be set according to the application scenario, etc. This embodiment does not impose any restrictions. In one optional embodiment, the bilinear kernel weights can be a table of size 32*32, as shown below:

[0221] static const uint8_t obmc_linear32

[1024] ={

[0222] 0，0，0，0，4，4，4，4，4，4，4，4，8，8，8，8，8，8，8，8，4，4，4，4，4，4，4，4，0，0，0，0，0，4，4，4，8，8，8，12，12，16，16，16，20，20，20，24，24，20，20，20，16，16，16，12，12，8，8，8，4，4，4，0，0，4，8，8，12，12，16，20，20，24，28，28，32，32，36，40，40，36，32，32，28，28，24，20，20，16，12，12，8，8，4，0，0，4，8，12，16，20，24，28，28，32，36，40，44，48，52，56，56，52，48，44，40，36，32，28，28，24，20，16，12，8，4，0，4，8，12，16，20，24，28，32，40，44，48，52，56，60，64，68，68，64，60，56，52，48，44，40，32，28，24，20，16，12，8，4，4，8，12，20，24，32，36，40，48，52，56，64，68，76，80，84，84，80，76，68，64，56，52，48，40，36，32，24，20，12，8，4，4，8，16，24，28，36，44，48，56，60，68，76，80，88，96，100，100，96，88，80，76，68，60，56，48，44，36，28，24，16，8，4，4，12，20，28，32，40，48，56，64，72，80，88，92，100，108，116，116，108，100，92，88，80，72，64，56，48，40，32，28，20，12，4，4，12，20，28，40，48，56，64，72，80，88，96，108，116，124，132，132，124，116，108，96，88，80，72，64，56，48，40，28，20，12，4，4，16，24，32，44，52，60，72，80，92，100，108，120，128，136，148，148，136，128，120，108，100，92，80，72，60，52，44，32，24，16，4，4，16，28，36，48，56，68，80，88，100，112，120，132，140，152，164，164，152，140，132，120，112，100，88，80，68，56，48，36，28，16，4，4，16，28，40，52，64，76，88，96，108，120，132，144，156，168，180，180，168，156，144，132，120，108，96，88，76，64，52，40，28，16，4，8，20，32，44，56，68，80，92，108，120，132，144，156，168，180，192，192，180，168，156，144，132，120，108，92，80，68，56，44，32，20，8，8，20，32，48，60，76，88，100，116，128，140，156，168，184，196，208，208，196，184，168，156，140，128，116，100，88，76，60，48，32，20，8，8，20，36，52，64，80，96，108，124，136，152，168，180，196，212，224，224，212，196，180，168，152，136，124，108，96，80，64，52，36，20，8，8，24，40，56，68，84，100，116，132，148，164，180，192，208，224，240，240，224，208，192，180，164，148，132，116，100，84，68，56，40，24，8，8，24，40，56，68，84，100，116，132，148，164，180，192，208，224，240，240，224，208，192，180，164，148，132，116，100，84，68，56，40，24，8，8，20，36，52，64，80，96，108，124，136，152，168，180，196，212，224，224，212，196，180，168，152，136，124，108，96，80，64，52，36，20，8，8，20，32，48，60，76，88，100，116，128，140，156，168，184，196，208，208，196，184，168，156，140，128，116，100，88，76，60，48，32，20，8，8，20，32，44，56，68，80，92，108，120，132，144，156，168，180，192，192，180，168，156，144，132，120，108，92，80，68，56，44，32，20，8，4，16，28，40，52，64，76，88，96，108，120，132，144，156，168，180，180，168，156，144，132，120，108，96，88，76，64，52，40，28，16，4，4，16，28，36，48，56，68，80，88，100，112，120，132，140，152，164，164，152，140，132，120，112，100，88，80，68，56，48，36，28，16，4，4，16，24，32，44，52，60，72，80，92，100，108，120，128，136，148，148，136，128，120，108，100，92，80，72，60，52，44，32，24，16，4，4，12，20，28，40，48，56，64，72，80，88，96，108，116，124，132，132，124，116，108，96，88，80，72，64，56，48，40，28，20，12，4，4，12，20，28，32，40，48，56，64，72，80，88，92，100，108，116，116，108，100，92，88，80，72，64，56，48，40，32，28，20，12，4，4，8，16，24，28，36，44，48，56，60，68，76，80，88，96，100，100，96，88，80，76，68，60，56，48，44，36，28，24，16，8，4，4，8，12，20，24，32，36，40，48，52，56，64，68，76，80，84，84，80，76，68，64，56，52，48，40，36，32，24，20，12，8，4，4，8，12，16，20，24，28，32，40，44，48，52，56，60，64，68，68，64，60，56，52，48，44，40，32，28，24，20，16，12，8，4，0，4，8，12，16，20，24，28，28，32，36，40，44，48，52，56，56，52，48，44，40，36，32，28，28，24，20，16，12，8，4，0，0，4，8，8，12，12，16，20，20，24，28，28，32，32，36，40，40，36，32，32，28，28，24，20，20，16，12，12，8，8，4，0，0, 4, 4, 4, 8, 8, 8, 12, 12, 16, 16, 16, 20, 20, 20, 24, 24, 20, 20, 20, 16, 16, 16, 12, 12, 8, 8, 8, 4, 4, 4, 0, 0, 0, 0, 0, 4, 4, 4, 4, 4, 4, 4, 4, 8, 8, 8, 8, 8, 8, 8, 8, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 0, 0, 0, 0,};,

[0223] The video processing method provided in this disclosure is robust to large motion scenes and can perform parallel computing to improve computational efficiency. For detailed areas such as dense textures, the optical flow is obtained more accurately, while the amount of computation is reduced, making it applicable to scenarios with limited computing power, such as mobile devices.

[0224] Furthermore, based on the above embodiments, in scenarios such as large limb movements, the optical flow obtained through iterative calculation may not converge, and in scenarios such as camera movement, the optical flow calculation at video frame boundaries may be inaccurate. Corresponding processing methods can be used to detect anomalies in the first and / or second optical flows, specifically including:

[0225] In one optional processing method, the accuracy of the first optical flow in complex scenarios such as large-amplitude limb movements can be improved by filtering out the first optical flow with abnormal values. Specifically, Figure 11 A flowchart illustrating yet another video processing method provided in this disclosure embodiment is shown below. Figure 11 As shown, it also includes:

[0226] Step 1101: Perform anomaly detection on the first optical flow of the first image block moving to the second video frame, and obtain the second image block corresponding to the first image block moving to the second video frame based on the first optical flow of the first image block to be detected.

[0227] In this embodiment, in order to improve the accuracy of the first optical flow when the first image block moves to the second video frame, anomaly detection is performed on the first optical flow.

[0228] In one optional implementation, taking the first optical flow of the first image block to be detected as an example, the second image block corresponding to the first optical flow in the second video frame can be determined based on the image block in the second video frame to which the endpoint of the first optical flow points. It should be noted that the first optical flow can be rounded down in this step.

[0229] Step 1102: Calculate the first offset vector between the first optical flow of the first image block to be detected and the second optical flow of the corresponding second image block in the second video frame, and compare the first offset vector with a preset first threshold.

[0230] After acquiring the second image block in the second video frame, the second optical flow of the second image block is acquired. A first offset vector is calculated between the first optical flow of the first image block to be detected and the second optical flow. This first offset vector can be used to characterize the difference between the first optical flow and the second optical flow. The vector length of the first offset vector is compared with a first threshold. The first threshold can be preset according to the preset requirements of the application scenario, and this embodiment does not impose any restrictions.

[0231] In one alternative implementation, the first offset vector may be the vector sum of the first optical flow and the second optical flow.

[0232] Step 1103: If the first offset vector is greater than the first threshold, then compare the vector length of the first optical flow of the first image block to be detected with the inverse vector length of the second optical flow of the corresponding second image block in the second video frame.

[0233] If the first offset vector is greater than the first threshold, it indicates that the first optical flow of the first image block may be abnormal, and further detection is required. This involves comparing the vector length of the first optical flow of the first image block with the vector length of the inverse vector of the second optical flow of the second image block. The inverse vector of the second optical flow can be a vector with the same length as the second optical flow but in the opposite direction.

[0234] Step 1104: If the inverse vector length of the second optical flow is less than the vector length of the first optical flow, then adjust the first optical flow of the first image block to be detected to the inverse vector of the second optical flow of the corresponding second image block in the second video frame.

[0235] If the vector length of the inverse vector of the second optical flow is less than the vector length of the first optical flow, in order to improve the accuracy of the first optical flow, the first optical flow of the first image block is adjusted to be the inverse vector of the second optical flow of the corresponding second image block. For example, if the vector length of the first optical flow is 4, the vector length of the second optical flow is 3, the vector length of the first offset vector between the first and second optical flows is 5, and the first threshold is 4, then the vector length 5 of the first offset vector is greater than the first threshold 4, and the vector length 3 of the inverse vector of the second optical flow is less than the vector length 4 of the first optical flow, then the first optical flow is adjusted to be the inverse vector of the second optical flow of the corresponding second image block.

[0236] In another optional processing method, the accuracy of the first optical flow in scenarios such as camera movement during the shooting of the video can be improved by processing the first optical flow of the first image block located at the boundary position in the first video frame. Figure 12 A flowchart illustrating yet another video processing method provided in this disclosure embodiment is shown below. Figure 12 As shown, specifically, it also includes:

[0237] Step 1201: Perform anomaly detection on the first image block corresponding to the row boundary or column boundary in the first video frame, and obtain the vector length corresponding to the first optical flow of the first image block of the row boundary or column boundary to be detected.

[0238] In this embodiment, anomaly detection is performed on the first image block corresponding to the row boundary or column boundary in the first video frame. The first image block corresponding to the row boundary of the first video frame can be the image block located in the outermost row of the first video frame, where the outermost row includes the top row and the bottom row. The first image block corresponding to the column boundary of the first video frame can also be the image block located in the outermost column of the first video frame, where the outermost column includes the leftmost column and the bottom column.

[0239] To determine whether the first optical flow of the first image block in the current row or column boundary to be detected is accurate, the vector length corresponding to the first optical flow is obtained.

[0240] Step 1202: Compare the vector length corresponding to the first optical flow of the first image block of the current row boundary or column boundary to be detected with a preset threshold value.

[0241] Then, the vector length of the first optical flow of the first image block included in the current row or column boundary to be detected is compared with a preset threshold value. This preset threshold value can be set according to the application scenario; this embodiment does not impose any restrictions. For example, the preset threshold value can be set to 0.

[0242] Step 1203: If the number of vector lengths less than the preset threshold is greater than the preset third threshold, then the first optical flow of the first image block of the current row boundary or column boundary to be detected is adjusted to the first optical flow of the first image block of the adjacent row or column of the current row boundary or column boundary to be detected.

[0243] Furthermore, the number of first optical flows with vector lengths less than a preset threshold in the currently detected row or column boundaries is counted. If this number is greater than a preset third threshold, if the current detection is a row boundary, the first optical flow of the first image block of that row boundary is adjusted to the first optical flow of the first image block of the adjacent row; if the current detection is a column boundary, the first optical flow of the first image block of the currently detected column boundary is adjusted to the first optical flow of the first image block of the adjacent column. The preset third threshold can be set according to the application scenario, and this embodiment does not impose any restrictions. For example, the preset third threshold can be set to 50% of the number of first image blocks at the row or column boundary.

[0244] For example, if the current line boundary to be detected is the topmost line in the first video frame, and the number of first image blocks in the topmost line is 50, the preset threshold value is 1, and the preset third threshold value is 25, and assuming that the number of vectors with a length of 0 in the first optical flow of the first image block in the topmost line is 30, and 30 is greater than the preset third threshold value of 25, then the first optical flow of the first image block in the topmost line is adjusted to the first optical flow of the first image block in the second line above the topmost line in the first video frame.

[0245] In one alternative processing method, the accuracy of the second optical flow can be improved in complex scenarios such as large-amplitude limb movements by filtering out the second optical flow with abnormal values. Figure 13 A flowchart illustrating yet another video processing method provided in this disclosure embodiment is shown below. Figure 13 As shown, specifically, it also includes:

[0246] Step 1301: Perform anomaly detection on the second optical flow of the second image block moving to the first video frame, and obtain the first image block corresponding to the second image block moving to the first video frame based on the second optical flow of the second image block to be detected.

[0247] In this embodiment, in order to improve the accuracy of the second optical flow when the second image block moves to the first video frame, anomaly detection is performed on the second optical flow.

[0248] In one optional implementation, taking the second optical flow of the second image block to be detected as an example, the first image block corresponding to the second optical flow in the first video frame can be determined based on the image block in the first video frame to which the endpoint of the second optical flow points. It should be noted that the first optical flow can be rounded down in this step.

[0249] Step 1302: Calculate the second offset vector between the second optical flow of the second image block to be detected and the first optical flow of the corresponding first image block in the first video frame, and compare the second offset vector with a preset second threshold.

[0250] After acquiring the first image block in the first video frame, the first optical flow of the first image block is acquired. Then, a second offset vector is calculated between the second optical flow of the second image block to be detected and the first optical flow. This second offset vector can be used to characterize the difference between the second optical flow and the first optical flow. The vector length of the second offset vector is compared with a second threshold. The second threshold can be preset according to the preset requirements of the application scenario, and this embodiment does not impose any restrictions.

[0251] In one alternative implementation, the second offset vector may be the vector sum of the second optical flow and the first optical flow.

[0252] Step 1303: If the second offset vector is greater than the second threshold, then compare the vector length of the second optical flow of the second image block to be detected with the inverse vector length of the first optical flow of the corresponding first image block in the first video frame.

[0253] If the second offset vector is greater than the second threshold, it indicates that the second optical flow of the second image block may be abnormal, and further detection is required. This involves comparing the vector length of the second optical flow of the second image block with the vector length of the inverse vector of the first optical flow of the first image block. The inverse vector of the first optical flow can be a vector with the same length as the first optical flow but in the opposite direction.

[0254] Step 1304: If the inverse vector length of the first optical flow is less than the vector length of the second optical flow, then adjust the second optical flow of the currently detected second image block to the inverse vector of the first optical flow of the corresponding first image block in the first video frame.

[0255] If the vector length of the inverse vector of the first optical flow is less than the vector length of the second optical flow, in order to improve the accuracy of the second optical flow, the second optical flow of the second image block is adjusted to the inverse vector of the first optical flow of the corresponding first image block.

[0256] For example, Figure 14 This is a schematic diagram illustrating the calculation of a second offset vector provided in an embodiment of the present disclosure, as shown below. Figure 14 As shown in the figure, mv 10 For the second optical flow of the second image block, in this example, it can be mv 10 Perform the floor function, mv in the diagram 01 For the first optical flow of the corresponding first image block, the second offset vector offset in the figure is relative to mv. 10 and MV 01 The vector sum is used; if the length of the second offset vector (offset) is greater than the second threshold, then mv is used. 10 Set as MV 10 and the inverse vector of the first optical flow -mv 01 The one with the smaller vector length.

[0257] Another optional processing method is to process the second optical flow of the second image block located at the boundary position in the second video frame, thereby improving the accuracy of the second optical flow in scenarios such as camera movement during the shooting of the video to be processed. Figure 15 A flowchart illustrating yet another video processing method provided in this disclosure embodiment is shown below. Figure 15 As shown, specifically, it also includes:

[0258] Step 1501: Perform anomaly detection on the second image block corresponding to the row boundary or column boundary in the second video frame, and obtain the vector length corresponding to the second optical flow of the second image block to be detected at the row boundary or column boundary.

[0259] In this embodiment, anomaly detection is performed on the second image blocks corresponding to the row boundaries or column boundaries in the second video frame. The second image block corresponding to the row boundary of the second video frame can be the image block located in the outermost row of the second video frame, where the outermost row includes the top row and the bottom row. The second image block corresponding to the column boundary of the second video frame can also be the image block located in the outermost column of the second video frame, where the outermost column includes the leftmost column and the bottom column.

[0260] To determine whether the second optical flow of the second image block in the current row or column boundary to be detected is accurate, the vector length corresponding to the second optical flow is obtained.

[0261] Step 1502: Compare the vector length corresponding to the second optical flow of the second image block at the current row or column boundary to be detected with a preset threshold value.

[0262] Then, the vector length of the second optical flow of the second image block included in the current row or column boundary to be detected is compared with a preset threshold value. This preset threshold value can be set according to the application scenario; this embodiment does not impose any restrictions. For example, the preset threshold value can be set to 0.

[0263] Step 1503: If the number of vector lengths less than the preset threshold is greater than the preset third threshold, then the second optical flow of the second image block at the current row or column boundary to be detected is adjusted to the second optical flow of the second image block in the adjacent row or column of the current row or column boundary to be detected.

[0264] Furthermore, the number of second optical flows with vector lengths less than a preset threshold in the currently detected row or column boundaries is counted. If this number is greater than a preset third threshold, and if the current detection is a row boundary, the second optical flow of the second image block of that row boundary is adjusted to the second optical flow of the second image block of the adjacent row; if the current detection is a column boundary, the second optical flow of the second image block of the currently detected column boundary is adjusted to the second optical flow of the second image block of the adjacent column. The preset third threshold can be set according to the application scenario, and this embodiment does not impose any restrictions. For example, the preset third threshold can be set to 50% of the number of second image blocks at the row or column boundary.

[0265] For example, if the column boundary to be detected is the leftmost column in the second video frame, and the number of second image blocks in the top row is 50, the preset threshold is 1, and the preset third threshold is 25, assuming that the number of vectors with a length of 0 in the second optical flow of the second image block in the leftmost column is 30, and 30 is greater than the preset third threshold of 25, then the second optical flow of the second image block in the leftmost column is adjusted to the second optical flow of the second image block in the leftmost second column adjacent to the leftmost column in the second video frame.

[0266] The video processing method provided in this disclosure can filter out optical flow with large errors, thereby improving the accuracy of optical flow calculation and ensuring the quality of video images.

[0267] Figure 16 This is a schematic diagram of a video processing apparatus provided in an embodiment of the present disclosure. The apparatus can be implemented by software and / or hardware and is generally integrated into an electronic device. Figure 16 As shown, the device includes:

[0268] The determining module 1601 is used to determine the first optical flow of a first image block moving from a first video frame to a second video frame, and the second optical flow of a second image block moving from a second video frame to the first video frame, wherein the first video frame and the second video frame are adjacent video frames, and the first image block and the second image block are image regions including multiple pixels.

[0269] The compositing module 1602 is used to compose an intermediate video frame based on the first video frame, the second video frame, the first optical flow, and the second optical flow, wherein the intermediate video frame is an estimated video frame to be inserted between the first video frame and the second video frame.

[0270] In one optional implementation, the determining module 1601 includes:

[0271] The scaling unit is used to scale the first video frame to obtain a corresponding first image set, and to scale the second video frame to obtain a corresponding second image set, wherein the first image set and the second image set include: multiple image layers with different resolutions;

[0272] The first computing unit is configured to start from the lowest resolution image layer in the first image set, calculate the initial optical flow of a pre-divided image block in the current layer image, and calculate the initial optical flow of a pre-divided image block in the next resolution image based on the initial optical flow of the image block in the current layer image, until the initial optical flow of the pre-divided image block in the highest resolution image layer is calculated, and determine it as the first optical flow of the first image block moving to the second video frame.

[0273] The second calculation unit is used to calculate the initial optical flow of a pre-divided image block in the current layer image, starting from the lowest resolution image layer in the second image set, and calculate the initial optical flow of a pre-divided image block in the next layer resolution image based on the initial optical flow of the image block in the current layer image, until the initial optical flow of the pre-divided image block in the highest resolution image layer is calculated, and determine it as the second optical flow of the second image block moving to the first video frame.

[0274] In one optional implementation, the first computing unit is configured to:

[0275] Obtain the first-direction gradient value and the second-direction gradient value of each pixel in the image block of the current layer image;

[0276] Based on the first and second gradient values ​​of each pixel, determine the first pixel matrix, the second pixel matrix, and the third pixel matrix corresponding to the image block in the current layer image;

[0277] The first pixel matrix, the second pixel matrix, and the third pixel matrix are processed according to a preset algorithm to obtain the initial optical flow corresponding to the image block in the current layer image.

[0278] In one optional embodiment, the apparatus further includes:

[0279] The first detection module is used to perform anomaly detection on the first optical flow of the first image block moving to the second video frame, and to obtain the second image block corresponding to the first image block moving to the second video frame based on the first optical flow of the first image block to be detected.

[0280] The first calculation module is used to calculate a first offset vector between the first optical flow of the current first image block to be detected and the second optical flow of the corresponding second image block in the second video frame, and compare the first offset vector with a preset first threshold.

[0281] The first processing module is configured to compare the vector length of the first optical flow of the first image block to be detected with the inverse vector length of the second optical flow of the corresponding second image block in the second video frame if the first offset vector is greater than the first threshold.

[0282] The second processing module is configured to adjust the first optical flow of the currently detected first image block to the inverse vector of the second optical flow of the corresponding second image block in the second video frame if the inverse vector length of the second optical flow is less than the vector length of the first optical flow.

[0283] In one optional embodiment, the apparatus further includes:

[0284] The second detection module is used to perform anomaly detection on the second optical flow of the second image block moving to the first video frame, and to obtain the first image block corresponding to the movement in the first video frame based on the second optical flow of the second image block to be detected.

[0285] The second calculation module is used to calculate the second offset vector between the second optical flow of the current second image block to be detected and the first optical flow of the corresponding first image block in the first video frame, and compare the second offset vector with a preset second threshold.

[0286] The third processing module is used to compare the vector length of the second optical flow of the currently detected second image block with the inverse vector length of the first optical flow of the corresponding first image block in the first video frame if the second offset vector is greater than the second threshold.

[0287] The fourth processing module is used to adjust the second optical flow of the currently detected second image block to the inverse vector of the first optical flow of the corresponding first image block in the first video frame if the inverse vector length of the first optical flow is less than the vector length of the second optical flow.

[0288] In one optional embodiment, the apparatus further includes:

[0289] The third detection module is used to perform anomaly detection on the first image block corresponding to the row boundary or column boundary in the first video frame, and to obtain the vector length corresponding to the first optical flow of the first image block to be detected at the row boundary or column boundary.

[0290] The fifth processing module is used to compare the vector length corresponding to the first optical flow of the first image block of the current row boundary or column boundary to be detected with a preset threshold value;

[0291] The sixth processing module is used to adjust the first optical flow of the first image block of the current row boundary or column boundary to the first optical flow of the first image block of the adjacent row or column of the current row boundary or column boundary if the number of vector lengths smaller than the preset threshold value is greater than the preset third threshold value.

[0292] The fourth detection module is used to perform anomaly detection on the second image block corresponding to the row boundary or column boundary in the second video frame, and to obtain the vector length corresponding to the second optical flow of the second image block to be detected at the row boundary or column boundary.

[0293] The seventh processing module is used to compare the vector length corresponding to the second optical flow of the second image block of the current row boundary or column boundary to be detected with a preset threshold value;

[0294] The eighth processing module is used to adjust the second optical flow of the second image block at the current row or column boundary to the second optical flow of the second image block at the current row or column boundary to the second optical flow of the second image block at the adjacent row or column of the current row or column boundary if the number of vector lengths less than the preset threshold value is greater than the preset third threshold value.

[0295] In one optional implementation, the synthesis module 1602 includes:

[0296] The acquisition unit is configured to perform motion search adjustment on the first optical flow of the first image block moving to the second video frame, acquire the third optical flow of the first image block moving to the second video frame, and perform motion search adjustment on the second optical flow of the second image block moving to the first video frame, acquire the fourth optical flow of the second image block moving to the first video frame.

[0297] The synthesis unit is configured to synthesize the intermediate video frame based on the first video frame, the second video frame, the third optical flow of the first image block moving to the second video frame, and the fourth optical flow of the second image block moving to the first video frame.

[0298] In one optional implementation, the acquisition unit is configured to:

[0299] Motion search is performed on the first image block to determine whether the first image block to be processed is located on the boundary of the first video frame. If the first image block to be processed is located on the boundary, no adjustment is made and the first optical flow of the first image block to be processed is used as the third optical flow to move to the second video frame.

[0300] If the first image block to be processed is not located at the boundary, then a first candidate vector array is established based on the first optical flow of the first image block to be processed, and the first candidate median of the first candidate vector array is determined.

[0301] Motion search is performed on the first image block within a first search vector range associated with the first candidate median, and a first target vector is determined within the first search vector range. The difference between the sum of all pixels of the image block in the second video frame corresponding to the first target vector and the sum of all pixels of the first image block to be processed is less than the difference between the sum of all pixels of the image block in the second video frame corresponding to other vectors within the first search vector range and the sum of all pixels of the first image block to be processed.

[0302] The first optical flow of the first image block to be processed is adjusted to the first target vector, which is then used as the third optical flow for the first image block to be processed to move to the second video frame.

[0303] In one optional implementation, the acquisition unit is configured to:

[0304] Motion search is performed on the second image block to determine whether the current second image block to be processed is located at the boundary of the second video frame. If the current second image block to be processed is located at the boundary, no adjustment is made and the second optical flow of the current second image block to be processed is used as the fourth optical flow to move to the first video frame.

[0305] If the current second image block to be processed is not located at the boundary, then a second candidate vector array is established based on the second optical flow of the current second image block to be processed, and the second candidate median of the second candidate vector array is determined.

[0306] Motion search is performed on the second image block within the second search vector range associated with the second candidate median, and a second target vector is determined within the second search vector range, wherein the difference between the sum of all pixels of the image block in the first video frame corresponding to the second target vector and the sum of all pixels of the currently to-be-processed second image block is less than the difference between the sum of all pixels of the image block in the first video frame corresponding to other vectors within the second search vector range and the sum of all pixels of the currently to-be-processed second image block.

[0307] The second optical flow of the currently unprocessed second image block is adjusted to the second target vector, which serves as the fourth optical flow for the currently unprocessed second image block to move to the first video frame.

[0308] In one optional implementation, the synthesis unit includes:

[0309] The first determining unit is configured to determine the coordinates of the first center point on the intermediate video frame corresponding to the first image block based on the third optical flow of the first image block moving to the second video frame and the insertion time of the intermediate video frame.

[0310] The second acquisition unit is used to sample and acquire a corresponding first sampling block on the first video frame according to the coordinates of each first center point, and to sample and acquire a corresponding second sampling block on the second video frame.

[0311] The first accumulation unit is used to accumulate the pixels of the first sampling block and the pixels of the second sampling block obtained according to each first center point coordinate to the intermediate video frame;

[0312] The second determining unit is used to determine the coordinates of the second center point corresponding to the second image block on the intermediate video frame based on the fourth optical flow of the second image block moving to the first video frame and the insertion time of the intermediate video frame;

[0313] The third acquisition unit is used to sample and acquire a corresponding third sampling block on the first video frame according to the coordinates of each second center point, and to sample and acquire a corresponding fourth sampling block on the second video frame.

[0314] The second accumulation unit is used to accumulate the pixels of the third sampling block and the fourth sampling block obtained corresponding to each second center point into the intermediate video frame.

[0315] In one optional embodiment, the apparatus further includes:

[0316] The third accumulation unit is used to accumulate the pixels of the first sampling block and the pixels of the second sampling block into the intermediate video frame according to the preset bilinear kernel weights, and to accumulate the pixels of the third sampling block and the pixels of the fourth sampling block into the intermediate video frame.

[0317] The video processing apparatus provided in this disclosure can execute the video processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0318] In addition to the methods and apparatus described above, this disclosure also provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to implement the video processing method described in this disclosure.

[0319] This disclosure also provides a computer program product, which includes a computer program / instructions that, when executed by a processor, implement the video processing method described in this disclosure.

[0320] Figure 17 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0321] The following is a detailed reference. Figure 17 The diagram illustrates a structural schematic suitable for implementing the electronic device 1700 in the embodiments of this disclosure. The electronic device 1700 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 17 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0322] like Figure 17As shown, electronic device 1700 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 1701, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1702 or a program loaded from storage device 1708 into random access memory (RAM) 1703. The RAM 1703 also stores various programs and data required for the operation of electronic device 1700. The processing unit 1701, ROM 1702, and RAM 1703 are interconnected via bus 1704. Input / output (I / O) interface 1705 is also connected to bus 1704.

[0323] Typically, the following devices can be connected to I / O interface 1705: input devices 1706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1709. Communication device 1709 allows electronic device 1700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 17 An electronic device 1700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0324] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1709, or installed from storage device 1708, or installed from ROM 1702. When the computer program is executed by processing device 1701, it performs the functions defined in the video processing method of embodiments of this disclosure.

[0325] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0326] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0327] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0328] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: determine a first optical flow from a first image block in a first video frame to a second video frame, and a second optical flow from a second image block in a second video frame to the first video frame, wherein the first and second video frames are adjacent video frames, and the first and second image blocks are image regions comprising multiple pixels; and synthesize an intermediate video frame based on the first video frame, the second video frame, the first optical flow, and the second optical flow, wherein the intermediate video frame is an estimated video frame to be inserted between the first video frame and the second video frame. It is evident that the embodiments of this disclosure improve the robustness and accuracy of video processing in scenes with large motion scales and reduce the computational load of estimating video frames, enabling video frame rate enhancement in application scenarios with limited computational resources, such as mobile devices.

[0329] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0330] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0331] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0332] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0333] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0334] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0335] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0336] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method of video processing, the method comprising: include: Determine the first optical flow of a first image block moving from a first video frame to a second video frame, and the second optical flow of a second image block moving from a second video frame to the first video frame, wherein the first video frame and the second video frame are adjacent video frames, and the first image block and the second image block are image regions including multiple pixels; An intermediate video frame is synthesized based on the first video frame, the second video frame, the first optical flow, and the second optical flow, wherein the intermediate video frame is an estimated video frame to be inserted between the first video frame and the second video frame; The step of synthesizing intermediate video frames based on the first video frame, the second video frame, the first optical flow, and the second optical flow includes: Motion search and adjustment are performed on the first optical flow of the first image block moving to the second video frame to obtain the third optical flow of the first image block moving to the second video frame, and motion search and adjustment are performed on the second optical flow of the second image block moving to the first video frame to obtain the fourth optical flow of the second image block moving to the first video frame. The intermediate video frame is synthesized based on the first video frame, the second video frame, the third optical flow of the first image block moving to the second video frame, and the fourth optical flow of the second image block moving to the first video frame.

2. The method of claim 1, wherein, The determination of the first optical flow from the first image block in the first video frame to the second video frame, and the second optical flow from the second image block in the second video frame to the first video frame, includes: The first video frame is scaled to obtain a corresponding first image set, and the second video frame is scaled to obtain a corresponding second image set, wherein the first image set and the second image set include: multiple image layers with different resolutions; Starting from the lowest resolution image layer in the first image set, calculate the initial optical flow of the pre-divided image blocks in the current layer image, and calculate the initial optical flow of the pre-divided image blocks in the next resolution image based on the initial optical flow of the image blocks in the current layer image, until the initial optical flow of the pre-divided image blocks in the highest resolution image layer is calculated, and determine it as the first optical flow of the first image block moving to the second video frame; Starting from the lowest resolution image layer in the second image set, the initial optical flow of the pre-divided image blocks in the current layer image is calculated, and the initial optical flow of the pre-divided image blocks in the next resolution image is calculated based on the initial optical flow of the image blocks in the current layer image, until the initial optical flow of the pre-divided image blocks in the highest resolution image layer is calculated, and determined as the second optical flow of the second image block moving to the first video frame.

3. The method of claim 2, wherein, The calculation of the initial optical flow of the pre-divided image blocks in the current layer image includes: Obtain the first-direction gradient value and the second-direction gradient value of each pixel in the image block of the current layer image; Based on the first and second gradient values ​​of each pixel, determine the first pixel matrix, the second pixel matrix, and the third pixel matrix corresponding to the image block in the current layer image; The first pixel matrix, the second pixel matrix, and the third pixel matrix are processed according to a preset algorithm to obtain the initial optical flow corresponding to the image block in the current layer image.

4. The method of claim 1, wherein, Also includes: Anomaly detection is performed on the first optical flow of the first image block moving to the second video frame, and the second image block corresponding to the first image block moving to the second video frame is obtained based on the first optical flow of the first image block to be detected. Calculate the first offset vector between the first optical flow of the current first image block to be detected and the second optical flow of the corresponding second image block in the second video frame, and compare the first offset vector with a preset first threshold. If the first offset vector is greater than the first threshold, then compare the vector length of the first optical flow of the first image block to be detected with the inverse vector length of the second optical flow of the corresponding second image block in the second video frame; If the inverse vector length of the second optical flow is less than the vector length of the first optical flow, then the first optical flow of the first image block to be detected is adjusted to the inverse vector of the second optical flow of the corresponding second image block in the second video frame.

5. The method of claim 1, wherein, Also includes: Anomaly detection is performed on the second optical flow of the second image block moving to the first video frame, and the first image block corresponding to the second image block moving to the first video frame is obtained based on the second optical flow of the second image block to be detected. Calculate the second offset vector between the second optical flow of the currently detected second image block and the first optical flow of the corresponding first image block in the first video frame, and compare the second offset vector with a preset second threshold. If the second offset vector is greater than the second threshold, then compare the vector length of the second optical flow of the currently detected second image block with the inverse vector length of the first optical flow of the corresponding first image block in the first video frame; If the inverse vector length of the first optical flow is less than the vector length of the second optical flow, then the second optical flow of the currently detected second image block is adjusted to the inverse vector of the first optical flow of the corresponding first image block in the first video frame.

6. The method of claim 1, wherein, Also includes: Anomaly detection is performed on the first image block corresponding to the row boundary or column boundary in the first video frame, and the vector length corresponding to the first optical flow of the first image block to be detected at the row boundary or column boundary is obtained. The vector length corresponding to the first optical flow of the first image block of the current row or column boundary to be detected is compared with a preset threshold value; If the number of vector lengths less than the preset threshold is greater than the preset third threshold, then the first optical flow of the first image block of the current row boundary or column boundary to be detected is adjusted to the first optical flow of the first image block of the adjacent row or column of the current row boundary or column boundary to be detected. And / or, Anomaly detection is performed on the second image block corresponding to the row boundary or column boundary in the second video frame, and the vector length corresponding to the second optical flow of the second image block to be detected at the row boundary or column boundary is obtained. The vector length corresponding to the second optical flow of the second image block at the current row or column boundary to be detected is compared with a preset threshold value; If the number of vector lengths less than the preset threshold is greater than the preset third threshold, then the second optical flow of the second image block at the current row or column boundary to be detected is adjusted to the second optical flow of the second image block in the adjacent row or column of the current row or column boundary to be detected.

7. The method of claim 1, wherein, The step of performing motion search adjustment on the first optical flow of the first image block moving to the second video frame, and obtaining the third optical flow of the first image block moving to the second video frame, includes: Motion search is performed on the first image block to determine whether the first image block to be processed is located on the boundary of the first video frame. If the first image block to be processed is located on the boundary, no adjustment is made and the first optical flow of the first image block to be processed is used as the third optical flow to move to the second video frame. If the first image block to be processed is not located at the boundary, then a first candidate vector array is established based on the first optical flow of the first image block to be processed, and the first candidate median of the first candidate vector array is determined. Motion search is performed on the first image block within a first search vector range associated with the first candidate median, and a first target vector is determined within the first search vector range. The difference between the sum of all pixels of the image block in the second video frame corresponding to the first target vector and the sum of all pixels of the first image block to be processed is less than the difference between the sum of all pixels of the image block in the second video frame corresponding to other vectors within the first search vector range and the sum of all pixels of the first image block to be processed. The first optical flow of the first image block to be processed is adjusted to the first target vector, which is then used as the third optical flow for the first image block to be processed to move to the second video frame.

8. The method of claim 1, wherein, The step of performing motion search adjustment on the second optical flow of the second image block moving to the first video frame, and obtaining the fourth optical flow of the second image block moving to the first video frame, includes: Motion search is performed on the second image block to determine whether the current second image block to be processed is located at the boundary of the second video frame. If the current second image block to be processed is located at the boundary, no adjustment is made and the second optical flow of the current second image block to be processed is used as the fourth optical flow to move to the first video frame. If the current second image block to be processed is not located at the boundary, then a second candidate vector array is established based on the second optical flow of the current second image block to be processed, and the second candidate median of the second candidate vector array is determined. Motion search is performed on the second image block within the second search vector range associated with the second candidate median, and a second target vector is determined within the second search vector range, wherein the difference between the sum of all pixels of the image block in the first video frame corresponding to the second target vector and the sum of all pixels of the currently to-be-processed second image block is less than the difference between the sum of all pixels of the image block in the first video frame corresponding to other vectors within the second search vector range and the sum of all pixels of the currently to-be-processed second image block. The second optical flow of the currently unprocessed second image block is adjusted to the second target vector, which serves as the fourth optical flow for the currently unprocessed second image block to move to the first video frame.

9. The method of claim 1, wherein, The step of synthesizing an intermediate video frame based on the first video frame, the second video frame, the third optical flow of the first image block moving to the second video frame, and the fourth optical flow of the second image block moving to the first video frame includes: Based on the third optical flow of the first image block moving to the second video frame and the insertion time of the intermediate video frame, determine the coordinates of the first center point on the intermediate video frame corresponding to the first image block; Based on the coordinates of each first center point, a corresponding first sampling block is obtained by sampling on the first video frame, and a corresponding second sampling block is obtained by sampling on the second video frame; The pixels of the first sampling block and the pixels of the second sampling block obtained according to the coordinates of each first center point are accumulated into the intermediate video frame; Based on the fourth optical flow of the second image block moving to the first video frame and the insertion time of the intermediate video frame, the coordinates of the second center point corresponding to the second image block on the intermediate video frame are determined; Based on the coordinates of each second center point, a corresponding third sampling block is obtained by sampling on the first video frame, and a corresponding fourth sampling block is obtained by sampling on the second video frame; The pixels of the third sampling block and the pixels of the fourth sampling block, obtained according to the coordinates of each second center point, are accumulated into the intermediate video frame.

10. The method of claim 9, wherein, Also includes: The pixels of the first sampling block and the second sampling block are accumulated into the intermediate video frame according to the preset bilinear kernel weights, and the pixels of the third sampling block and the fourth sampling block are accumulated into the intermediate video frame.

11. A video processing apparatus, comprising: The device includes: The determining module is used to determine the first optical flow of a first image block moving from a first video frame to a second video frame, and the second optical flow of a second image block moving from a second video frame to the first video frame, wherein the first video frame and the second video frame are adjacent video frames, and the first image block and the second image block are image regions including multiple pixels. A compositing module is used to compose an intermediate video frame based on the first video frame, the second video frame, the first optical flow, and the second optical flow, wherein the intermediate video frame is an estimated video frame to be inserted between the first video frame and the second video frame; The step of synthesizing intermediate video frames based on the first video frame, the second video frame, the first optical flow, and the second optical flow includes: Motion search and adjustment are performed on the first optical flow of the first image block moving to the second video frame to obtain the third optical flow of the first image block moving to the second video frame, and motion search and adjustment are performed on the second optical flow of the second image block moving to the first video frame to obtain the fourth optical flow of the second image block moving to the first video frame. The intermediate video frame is synthesized based on the first video frame, the second video frame, the third optical flow of the first image block moving to the second video frame, and the fourth optical flow of the second image block moving to the first video frame.

12. An electronic device, comprising: The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the video processing method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to implement the video processing method as described in any one of claims 1-10.

14. A computer program product, characterised in that, The computer program product includes a computer program / instruction that, when executed by a processor, implements the video processing method as described in any one of claims 1-10.

Images