Video frame image motion estimation method and related device

Abstract

The embodiment of the present application discloses a kind of motion estimation method of video frame image and related equipment, and wherein method includes: obtaining the first frame image in target video, first frame image is divided into multiple image blocks;Obtain the first center image block in multiple image blocks;First center image block is used to carry out rough search for multiple first candidate image blocks in first candidate search set, and determine second center image block according to rough search result;Second center image block is used to determine multiple second candidate image blocks in second candidate search set;First center image block is used to carry out accurate search for multiple second candidate image blocks, and determine the motion estimation result of first center image block according to accurate search result;The motion estimation result of first frame image is determined according to the motion estimation result of multiple image blocks of first frame image.The embodiment of the present application improves the overall efficiency and accuracy of motion estimation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video frame interpolation, and in particular to a video frame image motion estimation method and related equipment. BACKGROUND

[0002] In the motion estimation process of video frame interpolation, a video frame image is first divided into non-overlapping image blocks, for each image block in the current frame, a matching block most similar to the image block in the current frame is found in a certain search range in the reference frame, and then a motion vector is calculated.

[0003] There are various motion estimation algorithms. The optical flow method is to calculate the motion vector of each pixel point independently to obtain the optical flow field for motion estimation. The pixel recursive method updates the prediction value by recursion for each pixel to obtain the motion vector. The block matching-based motion estimation algorithm searches for image blocks of adjacent frames in the image through the physical distance between pixels, and finds the best result according to the matching rule. The simplest block matching algorithm is full search, which searches for matching blocks in a given search range, such as the entire image. The 3-Dimension Recursive Search (3DRS) method is a block matching algorithm, which inherits the motion vector of the adjacent block for the current block, calculates the cost value of the block corresponding to each candidate vector according to the matching criterion, and finds the most similar block by comparing the cost value.

[0004] However, using the 3DRS algorithm for motion estimation of video frame images has the problems of small search range, not considering the overall motion of the video, not performing accurate search, etc. Combining the 3DRS algorithm with other existing algorithms for motion estimation also has the problems of complex calculation, long time consumption for motion estimation, etc. SUMMARY

[0005] The embodiments of the present application provide a video frame image motion estimation method and related equipment, which can increase the search range of motion estimation, without the need for a large amount of calculation, and improve the overall efficiency and accuracy of motion estimation.

[0006] In a first aspect, an embodiment of the present application provides a motion estimation method for video frame images, which comprises: obtaining a first frame image in a target video, and dividing the first frame image into a plurality of image blocks; obtaining a first center image block in the plurality of image blocks; performing coarse search on a plurality of first candidate image blocks in a first candidate search set by using the first center image block, and determining a second center image block according to a coarse search result, wherein the first candidate image blocks are determined according to image blocks in a neighboring frame image of the first frame image; determining a plurality of second candidate image blocks in a second candidate search set according to the second center image block, wherein the second candidate image blocks are determined according to a step distance between the second center image block and other image blocks, and the other image blocks are located in the same frame image as the second center image block; performing accurate search on the plurality of second candidate image blocks by using the first center image block, and determining a motion estimation result of the first center image block according to an accurate search result; and determining a motion estimation result of the first frame image according to motion estimation results of the plurality of image blocks in the first frame image.

[0007] It can be seen that, in the embodiment of the present application, when performing motion estimation on image blocks in a video frame image, full search is used to perform motion estimation on each image block in the frame image, and for the search process of each image block, two searches are performed, i.e., coarse search and accurate search, the first candidate search set of the coarse search is determined according to image blocks in a neighboring frame image of the frame image on which motion estimation is currently performed, and the second candidate search set of the accurate search is obtained based on the result of the coarse search, which is an image block within a preset step. This process expands the search range through two searches and improves the search result. In addition, the process of obtaining the search set is simple in calculation, which improves the efficiency of the search process.

[0008] In a second aspect, an embodiment of the present application provides a motion estimation device, which comprises: an obtaining module configured to obtain a first frame image in a target video; and a processing module configured to divide the first frame image into a plurality of image blocks, and obtain a first center image block in the plurality of image blocks; the processing module is further configured to perform coarse search on a plurality of first candidate image blocks in a first candidate search set by using the first center image block, and determine a second center image block according to a coarse search result, wherein the first candidate image blocks are determined according to image blocks in a neighboring frame image of the first frame image; the processing module is further configured to determine a plurality of second candidate image blocks in a second candidate search set according to the second center image block, wherein the second candidate image blocks are determined according to a step distance between the second center image block and other image blocks, and the other image blocks are located in the same frame image as the second center image block; the processing module is further configured to perform accurate search on the plurality of second candidate image blocks by using the first center image block, and determine a motion estimation result of the first center image block according to an accurate search result; and determine a motion estimation result of the first frame image according to motion estimation results of the plurality of image blocks in the first frame image.

[0009] In a third aspect, an embodiment of the present application provides a motion estimation device, comprising: a processor and a memory;

[0010] The processor and the memory are connected, wherein the memory is configured to store program code, and the processor is configured to invoke the program code to execute the motion estimation method of the video frame image according to the first aspect.

[0011] In a fourth aspect, an embodiment of the present application provides a chip system, comprising: a processor and a memory coupled to the processor, wherein the memory is configured to store program or instructions, and when the program or instructions are executed by the processor, the chip system implements the method in the first aspect or any possible implementation manner of the first aspect.

[0012] In a fifth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program, and the computer program comprises program instructions, and when the program instructions are executed by a processor, the motion estimation method of the video frame image according to the first aspect is executed.

[0013] In a sixth aspect, an embodiment of the present application provides a computer program product, and when a computer reads and executes the computer program product, the computer executes the method in the first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0015] Figure 1 A full search matching method provided by an embodiment of the present application is shown in the figure;

[0016] Figure 2A A video frame image motion estimation method flowchart provided by an embodiment of the present application is shown in the figure;

[0017] Figure 2B A first frame image image block division diagram provided by an embodiment of the present application is shown in the figure;

[0018] Figure 2C A process diagram for obtaining a plurality of first candidate image blocks provided by an embodiment of the present application is shown in the figure;

[0019] Figure 2D A motion vector clustering flowchart provided by an embodiment of the present application is shown in the figure;

[0020] Figure 2E A process for determining a plurality of second candidate image blocks provided by an embodiment of the present application;

[0021] Figure 3 A structure of a motion estimation device provided by an embodiment of the present application;

[0022] Figure 4 A structure of a motion estimation device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described in connection with the drawings in the embodiments of the present application.

[0024] It should be understood that the terms "first", "second", etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, and are not used to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0025] In the present application, "embodiment" means that the specific features, structures or characteristics described in connection with the embodiment can be included in at least one embodiment of the present application. The phrase appears at various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present application can be combined with other embodiments.

[0026] First, the professional terms that can be involved in the embodiments of the present application are introduced.

[0027] Motion Estimation (ME): divide an image into a plurality of non-overlapping image blocks, and consider that the displacement amount of all pixels in the image block is the same, find the most similar image block to each image block in a given search range of a reference frame according to a certain matching criterion, i.e. the matching block, and obtain the relative displacement amount of the spatial positions of the two image blocks.

[0028] Motion Vector (MV): the relative displacement amount of the matching block and the current block (the image block used for matching search) is the motion vector.

[0029] Motion Vector Field: the motion vectors of all image blocks in an image constitute a motion vector field.

[0030] Motion compensation: the process of reconstructing an intermediate frame that does not exist originally by using the original frames and motion information.

[0031] Video interpolation: the process of generating new frames in a video sequence by motion estimation and motion compensation, etc. to improve the temporal resolution of the video.

[0032] Interpolated frame: a new frame generated by video interpolation between two frames of a video sequence.

[0033] The video sequence interpolation process based on motion estimation and motion compensation needs to use the information of the adjacent frames before and after to estimate the motion of the interpolated frame relative to them. The quality of the interpolated frame depends on the accuracy of the motion estimation. In order to get a good quality interpolated frame, a motion estimation algorithm that can get a real motion vector field needs to be selected. Some existing motion estimation algorithms have high computational complexity and are not easy to implement, such as optical flow field. The full search method is the simplest block matching algorithm, which searches for matching blocks in a given search range, such as the entire image, and then obtains the motion vector. As shown in FIG. 1, the search algorithm has high precision but the calculation amount is too large; searching for matching blocks in a limited search range reduces the computational load, but the given image block cannot well represent the real motion of the object. Figure 1

[0034] Based on the above description, please refer to Figure 2A , Figure 2A A flowchart of a video frame image motion estimation method is provided for the embodiments of the present application, as shown in FIG. 2, which includes the following steps: Figure 2A

[0035] 101. Obtain a first frame image in a target video, and divide the first frame image into a plurality of image blocks;

[0036] 102. Obtain a first center image block in the plurality of image blocks;

[0037] 103. Perform a rough search on a plurality of first candidate image blocks in a first candidate search set using the first center image block, and determine a second center image block according to the rough search result, wherein the first candidate image blocks are determined according to image blocks in a frame image adjacent to the first frame image;

[0038] 104. Determine a plurality of second candidate image blocks in a second candidate search set according to the second center image block, wherein the second candidate image blocks are determined according to the step distance between the second center image block and other image blocks, and the other image blocks are located in the same frame image as the second center image block;

[0039] ​​105. Use the first center image block to perform a precise search for multiple second candidate image blocks, and determine the motion estimation result of the first center image block based on the precise search results;

[0040] 106. Determine the motion estimation result of the first frame image based on the motion estimation results of multiple image blocks in the first frame image.

[0041] In this embodiment of the application, the target video is a video that needs to be interpolated. The first frame image in the target video image is obtained. The first frame image is a frame image with an unknown motion direction, and video interpolation is attempted after the first frame image. That is, assuming that the second frame image is the next frame image adjacent to the first frame image, in order to determine how to perform video interpolation between the first frame image and the second frame image, it is necessary to perform motion estimation on the first frame image. Specifically, the motion vector corresponding to the motion of the first frame image to the second frame image is obtained.

[0042] Divide the first frame image into multiple image blocks, please refer to Figure 2B , Figure 2B This is a schematic diagram illustrating the division of a first frame image into image blocks, as provided in an embodiment of this application. Figure 2B As shown in (a), the first frame image can be divided into multiple image blocks of the same size and shape, or, as... Figure 2B Option (b) could also involve dividing the first frame image into image blocks of different sizes based on pixel values. For example, smaller image blocks could be created for areas with larger pixel values ​​(darker colors) in the frame image, resulting in a more detailed division, while larger image blocks could be created for areas with smaller pixel values ​​(lighter colors), resulting in a coarser division. This division method is based on the general assumption that darker colors generally have more detail, thus requiring more precise matching.

[0043] In this embodiment, a method for dividing a first frame image into multiple image blocks of the same size and shape is used as an example. The first frame image is divided into several rectangles of the same size according to a preset size, and each rectangle corresponds to an image block. Any one of these image blocks can be selected as the first center image block for motion estimation. Figure 2B As described in (a), image block C is the first central image block selected in the embodiments of this application.

[0044] Then the first center image block is used to search and match image blocks in a reference frame to determine a matching block with the highest matching degree with the first center image block, to calculate a motion vector between the first center block and the matching block, and to complete motion estimation of the first center image block according to the motion vector. The reference frame can be a forward reference frame or a backward reference frame. The forward reference frame refers to an image frame corresponding to a previous time of a current time of a current frame (an image frame currently requiring motion estimation). The backward reference frame refers to an image frame to which the current frame can possibly move at a next time. The search and matching of the current frame and the forward reference frame refers to motion estimation for the previous time of the current time. The search and matching of the current frame and the backward reference frame refers to motion estimation for the next time of the current time. In the embodiments of the present application, the search and matching with the backward reference frame is taken as an example for description of motion estimation.

[0045] In the embodiments of the present application, the first center image block is used to search and match image blocks in a reference frame (a next frame image adjacent to the first frame image), and the search and matching specifically includes: using the first center image block to perform coarse search on a plurality of first candidate image blocks in a first candidate search set, determining a second center image block according to a coarse search result, wherein the first candidate image block is determined according to an image block in an adjacent frame image of the first frame image; using the second center image block to perform accurate search on a plurality of second candidate image blocks in a second candidate search set, determining a motion estimation result of the first frame image according to an accurate search result, wherein the second candidate image block is determined according to a step distance between the second center image block and other image blocks, and the other image blocks are located in the same frame image as the second center image block.

[0046] According to the above description, in the process of searching and matching the first center image block with image blocks in the reference frame, two searches are required. The first search is coarse search, which is used to determine the second center image block. The second search is accurate search, which is used to finally determine the motion estimation result. The coarse search is actually a matching operation of the first center image block with a plurality of first candidate image blocks in the first candidate search set, and then a second center image block is determined according to the matching result. The second center image block can be a matching block with the highest matching degree with the first center image block in the first candidate image block, or an image block obtained by adjusting the matching block. The first candidate image block is determined according to an image block in an adjacent frame image of the first frame image. For example, the first candidate image block can be an image block corresponding to an adjacent image block of the first center image block in the adjacent frame image. The adjacent frame image can be a previous frame image or a next frame image of the first frame image. The corresponding image block refers to an image block corresponding to the same pixel image on different frame images at different times. Or the first candidate image block can be an image block corresponding to the first center image block possibly on the next frame image, which is predicted according to a motion vector of the previous frame image block.

[0047] After the second center image block is determined according to the rough search result, a plurality of second candidate image blocks in a second candidate search set are determined according to the second center image block, the second center image block being an image block on a next frame image (which can be named as a second frame image) adjacent to the first frame image, that is, after the rough search, it is determined that the first center image block can be located at the position of the second center image block when the first frame image is transformed to the second frame image at the next time. Further, the second center image block can not be the image block with the highest matching degree with the first center image block on the second frame image, therefore, a plurality of second candidate image blocks on the second frame image can be further obtained, and then the matching block with the highest matching degree with the first center image block is obtained from the second candidate image blocks, and it is determined that the first center image block can be finally located at the position when the first frame image is transformed to the second frame image at the next time.

[0048] Each image block in the first frame image can be selected as the first center image block, then the motion estimation is performed by using the above method, and finally the motion estimation result of the first frame image is determined according to the motion vector of each image block, and then the position for video interpolation is determined.

[0049] It can be seen that in the embodiment of the present application, the first frame image in the target video image is obtained, and then the first center image block in the first frame image is obtained, the rough search and the accurate search are performed on the first center image block, wherein the image block used for matching search in the rough search is determined according to the image block in the adjacent frame image of the first frame image, the image block used for matching search in the accurate search is determined according to the second center image block determined by the rough search and the matching result of the image block within the preset step range and the second center image block, finally the image block with the highest matching degree with the first center image block is obtained as the final matching block, and then the motion estimation result of the first center image block is determined. The process expands the search range by two times of search, and improves the search result. In addition, the process of obtaining the candidate image block is simple in calculation, and the efficiency of the search process is improved.

[0050] Optionally, the method further includes determining a plurality of first candidate image blocks in the first candidate search set. For details, please refer to Figure 2C , Figure 2C A process diagram for obtaining a plurality of first candidate image blocks provided by the embodiment of the present application is shown in (b) of FIG. 1. Figure 2C n The first frame image is also the current frame, and the image block C therein is the obtained first center image block. First, the current frame f n ​Above, the first image block S1 and the second image block S2 to the left and above image block C have completed motion estimation search, and are spatially closer to image block C than other blocks. Therefore, the prediction vectors of S1 and S2 are selected as the spatial prediction vectors. That is to say, if... Figure 2C As shown in (c), the predicted motion vectors corresponding to S1 and S2 are used to calculate and obtain their respective values ​​in the second frame image f. n+1 The third image block S1' and the fourth image block S2' on the second frame are selected as the two first candidate image blocks. n+1 To be with f n The next adjacent frame image.

[0051] Other examples Figure 2C As shown in (a) in the figure, f n-1 It is the current frame f n The previous frame image is called the third frame image, and f is selected. n-1 The fifth image block T1 to the right and the sixth image block T2 to the bottom of the corresponding image block C' are used as the temporal prediction vector, where image block C' is f n Image block C in the previous frame corresponds to the image block in the image. Assuming that image block C has the same motion vector as image blocks T1 and T2, based on the continuity of object motion in the image, then T1 and T2 in frame f... n The corresponding image blocks are T1' and T2', which are also the image blocks adjacent to the right and below image block C. T1' and T2' retain the motion vectors of T1 and T2 from the previous frame, and their position in f can be obtained. n+1 The corresponding seventh image block T1” and eighth image block T2” are also considered as two other first candidate image blocks.

[0052] Additionally, image block C is obtained in f n+1 The image patch Zero corresponding to the same position in f is taken as the zero-point image patch. Same position refers to the image patch Zero at position f. n+1 The coordinate position of the image patch C in f n The coordinates are the same. The zero-point image patch is also a first candidate image patch.

[0053] For regions with complex motion, additional spatiotemporal image patches can be added to improve frame interpolation quality. Therefore, some global motion vectors in the temporal domain can be added as a supplement to the first candidate search set.

[0054] The global motion vector is mainly used to compensate for the fact that for regions with complex motion, searching using the first candidate image block determined by the adjacent image blocks of the first center image block may not capture the necessary variables. The global motion vector corresponding to the previous frame image is used in the coarse search process of the first center image block.

[0055] The global motion vector represents the motion vector of each image block in the whole frame image. After classification by a certain method, the motion vector of the category with a large proportion is obtained. In order to find the motion vector of most image blocks, the motion vector in the previous frame image can be classified.

[0056] For details, please refer to Figure 2D , Figure 2D A motion vector clustering flowchart provided in the embodiments of the present application is shown in FIG. 1. The global motion vector acquisition process of the third frame image (the previous frame image of the first frame image) includes the following steps: Figure 2D

[0057] 201, initialize the center of the class as a zero vector, the number of classes k = 1, select the class x = 0, and set a distance D;

[0058] 202, determine whether the selected class number x is less than 4;

[0059] 203, if yes, calculate the distance d between the motion vector in a row of image blocks and the center vector of the existing class, and compare the distance d with the distance D;

[0060] 204, if the distance d is less than or equal to the distance D, the motion vector of the image block is divided into a class, and after adding a new motion vector, the number of motion vectors in the class is +1, which is represented as count++;

[0061] 205, if the distance d is greater than the distance D, the motion vector of the image is classified into a new class, and the number of classes k++; determine whether the total number of classes k is less than K, if yes, execute step 206; if no, stop clustering;

[0062] 207, add a new motion vector to a class, recalculate the average value of the motion vector in the current class, and update the average value as the new class center vector mv_c, and execute step 202;

[0063] 208, after clustering, the number of motion vectors count in the class exceeds 1 / 8 of the number of image blocks in each row of the frame image, and the target class in the top four of the number of motion vectors is obtained;

[0064] 209, obtain the target center vector mv_c of the target class, and determine the global image block on the first center image block according to the target center vector. n+1

[0065] ​​According to the above description, the motion vectors of each image block of the third frame image are clustered to obtain a plurality of classification categories, and the total number of the plurality of classification categories cannot be greater than K. In the embodiment of the present application, K can be maximally set to 16. In some cases, if the third frame image is a small area image, K can be a value less than 16. The target class whose motion vector number is more than 1 / 8 of the number of image blocks per row of the third frame image and whose motion vector number is in the top four is selected, and the number of image blocks per row is the number of image blocks transversely divided in the third frame image. After obtaining the target class in the top four, the center vectors of the four target classes can be obtained, and according to the four center vectors, the four global image blocks corresponding to the first center image block on the second frame image can be calculated and obtained, which are also the image blocks in the first candidate search set.

[0066] According to the above description, the first candidate search set includes a total of 9 first candidate image blocks. After obtaining all the first candidate image blocks in the first candidate search set, the first center image block and the plurality of first candidate image blocks are respectively matched (i.e., rough search) to obtain the image block with the highest matching degree with the first center image block as the second center image block. The matching algorithm between images includes a gray-based matching algorithm and a feature-based matching algorithm, and the feature-based matching method includes a median absolute deviation (MAD) algorithm, a sum of absolute differences (SAD) algorithm, a sum of squared difference (SSD) algorithm, etc. In the embodiment of the present application, the SAD algorithm is used to determine the matching degree of the first center image block and the first candidate image block by calculating the sum of the absolute values of the pixel value differences in each corresponding pixel block, and the larger the SAD value is, the lower the matching degree of the two image blocks is. This process has the advantage of low computational complexity and can ensure high rough search efficiency. Finally, the image block with the highest matching degree with the first center image block in the first candidate image block is obtained as the second center image block.

[0067] In an optional case, in order to give a weighted average to the continuously obtained actual data and the original predicted data to make the prediction result closer to the actual situation, when calculating the matching degree of the first candidate image block and the first center image block, the sum of the absolute values of the pixel value differences between the two (the first absolute error sum) is obtained according to the SAD algorithm, and a smoothness term smoothness1 is added, that is, the first absolute error sum and the smoothness term are summed to obtain a first estimated value, and the smoothness term is specifically:

[0068] smoothness1 = ∑ |mv c1 -mvneighbor | (1)

[0069] wherein mv c1 represents the motion vector of the first candidate image block, mv neighbor represents the motion vectors of the eight neighboring image blocks of the first center image block on the first frame image, wherein the motion vectors of S1 and S2 are spatial prediction vectors, the motion vectors of T1 and T2 are temporal prediction vectors, and the motion vector of Zero is a zero vector.

[0070] Finally, the matching degree between the first center image block and the first candidate image block is determined according to the first estimation value, and the greater the first estimation value, the lower the matching degree.

[0071] It can be seen that in the embodiments of the present application, when the first center image block is selected to perform a rough search on the multiple first candidate image blocks in the first candidate search set, the image blocks obtained by combining the temporal prediction vectors with the neighboring image blocks of the first center image block, the image blocks obtained by combining the spatial prediction vectors with the neighboring image blocks of the first center image block, the image blocks corresponding to the same position on the next frame image of the first center image block, and the global image blocks determined according to the global motion vector of the previous frame image are all considered. This process fully considers the candidate image blocks that the neighboring image blocks of the first center image block may correspond to in various cases, and also considers the global motion vector of the previous frame image, thereby improving the representativeness and comprehensiveness of the obtained first candidate search set, and further improving the reliability of the rough search result.

[0072] After the rough search is completed, a precise search is further performed. As known from the foregoing process, the multiple second candidate image blocks in the second candidate search set used in the precise search are determined according to the step distance between the second center image block and other image blocks. The second center image block and the other image blocks are all located on the second frame image, and the step distance represents the straight-line distance between the image blocks. The step distance between neighboring image blocks is 1.

[0073] Specifically, please refer to Figure 2E , Figure 2E A process diagram for determining multiple second candidate image blocks provided by the embodiments of the present application, the second center image block is image block C0, as shown in Figure 2EAs shown in (a), an image block with a step distance of 1 from the second center image block is selected as the first step-length image block. The first distance can be, for example, 1, meaning that an image block with a step distance of 1 from the second center image block is selected as the first step-length image block. Specifically, this is the image block marked as 1 in the figure, which includes 8 image blocks adjacent to image block C0. Then, these 8 image blocks are matched with image block C0 respectively. The image matching algorithm described above can also be used to obtain the first step-length image block with the highest matching degree with image block C0, which is the third center image block 1-C0. Then, an image block with a step distance of 1 from image block 1-C0 is selected as the second step-length image block, which is the image block marked as 1' in the figure. In fact, the second step-length image block overlaps with the first step-length image block. The overlapping part is used as the first step-length image block and is not recorded again as the second step-length image block.

[0074] Then, as Figure 2E As shown in (b), image blocks with a step distance of 3 from the second center image block are selected as the third step-size image blocks. The second distance can be 3, meaning image blocks with a step distance of 3 from the second center image block are selected as the third step-size image blocks. Specifically, these are the 8 image blocks marked as 3 in the figure. These 8 image blocks are then matched with image block C0 to obtain the third step-size image block with the highest matching degree to image block C0, which is designated as the fourth center image block 3-C0. Then, image blocks with a step distance of 3 from image block 3-C0 are selected as the fourth step-size image blocks, which are the image blocks marked as 3' in the figure. Similarly, the fourth step-size image blocks contain image blocks that overlap with the third step-size image blocks; the overlapping parts are used as the third step-size image blocks and are not repeatedly recorded as the fourth step-size image blocks.

[0075] The second central image block obtained using the above method, the first step-length image block, the second step-length image block, the third step-length image block, and the fourth step-length image block constitute multiple second candidate image blocks in the second candidate search set. Then, the first central image block is used to perform a precise search on the multiple second candidate image blocks, that is, image matching is performed between the second central image block and each image block in the multiple second candidate image blocks. The matching method used can be, for example, the grayscale-based matching algorithm or the feature-based matching algorithm described above. In particular, the SAD algorithm can be used to improve matching efficiency. Finally, the image block with the highest matching degree to the first central image block among the multiple second candidate image blocks is obtained, and this is called the final matching block.

[0076] Optionally, when calculating the matching degree between the second candidate image block and the first center image block, the sum of the absolute values of the difference between the pixel values of the two (the second absolute error sum) is obtained according to the SAD algorithm, and a smoothness term smoothness2 and a distance difference distance can also be added, that is, the second absolute error sum, the smoothness term and the distance difference are summed to obtain a second estimation value, and the corresponding formula is:

[0077] smoothness2 = ∑ |mv c2 -mv neighbor | (2)

[0078] distance = max (|x|, |y|) (3)

[0079] wherein mv c2 represents the motion vector of the second candidate image block, mv neighbor represents the motion vector of the eight adjacent image blocks of the first center image block in the first frame image, and the motion vector of the second candidate image block can also be a temporal prediction vector or a spatial prediction vector. The distance difference (the first distance difference) is the maximum value of the absolute values of the x or y direction offsets of the motion vectors between the first center image block and the second candidate image block.

[0080] Finally, the matching degree between the first center image block and the second candidate image block is determined according to the second estimation value, and the greater the second estimation value, the lower the matching degree.

[0081] It can be seen that in the embodiment of the application, when the first center image block is selected for accurate search of the plurality of second candidate image blocks in the corresponding second candidate search set, the second step image block and the fourth step image block are obtained according to the first distance and the second distance, respectively, and then the accurate search result is obtained according to the matching results of the first center image block and the second step image block and the fourth step image block. This process can further increase the search range in the motion estimation process, increase the number of search image blocks, and the process of obtaining the candidate image block is simple, thereby improving the efficiency and accuracy of motion estimation.

[0082] Based on the description of the above-mentioned embodiment of the motion estimation method of the video frame image, the embodiment of the application also discloses a motion estimation device, which refers to Figure 3 , Figure 3 is a structural schematic diagram of a motion estimation device provided by the embodiment of the application. The motion estimation device 300 comprises:

[0083] The acquisition module 301 is configured to acquire a first frame image in a target video.

[0084] The processing module 302 is configured to divide the first frame image into a plurality of image blocks, and obtain a first center image block in the plurality of image blocks.

[0085] The processing module 302 is further configured to perform a coarse search on the plurality of first candidate image blocks in the first candidate search set using the first center image block, and determine a second center image block according to a result of the coarse search, wherein the first candidate image blocks are determined according to image blocks in a neighboring frame image of the first frame image;

[0086] The processing module 302 is further configured to determine a plurality of second candidate image blocks in a second candidate search set according to the second center image block, wherein the second candidate image blocks are determined according to a step distance between the second center image block and other image blocks in the same frame image as the second center image block.

[0087] The processing module 302 is further configured to perform a fine search on the plurality of second candidate image blocks using the first center image block, and determine a motion estimation result of the first center image block according to a result of the fine search; and determine a motion estimation result of the first frame image according to motion estimation results of the plurality of image blocks in the first frame image.

[0088] It can be seen that, in the embodiments of the present application, when performing motion estimation on image blocks in a frame image, a full search method is used to perform motion estimation on each image block in the frame image, and for the search process of each image block, two searches are performed, i.e., a coarse search and a fine search. The first candidate search set for the coarse search is determined according to image blocks in a neighboring frame image of the frame image on which motion estimation is currently performed, and the second candidate search set for the fine search is obtained based on a result of the coarse search. The process of the two searches expands the search range and improves the search result. In addition, the process of obtaining the search set is simple, and the efficiency of the search process is improved.

[0089] Optionally, the processing module 302 is further configured to determine the plurality of first candidate image blocks in the first candidate search set, and specifically configured to:

[0090] obtain a first motion vector of the first image block and a second motion vector of the second image block, and determine a third image block corresponding to the first image block when the first frame image moves to a second frame image according to the first image block and the first motion vector, and determine a fourth image block corresponding to the second image block when the first frame image moves to the second frame image according to the second image block and the second motion vector, wherein the first image block is a left neighboring image block of the first center image block, the second image block is an upper neighboring image block of the first center image block, and the second frame image is a next frame image adjacent to the first frame image.

[0091] the third motion vector of the fifth image block moving from a position on the third frame image to a current position, and the fourth motion vector of the sixth image block moving from a position on the third frame image to the current position, and determining a seventh image block corresponding to the fifth image block when the first frame image moves to the second frame image according to the fifth image block and the third motion vector, and determining an eighth image block corresponding to the sixth image block when the first frame image moves to the second frame image according to the sixth image block and the fourth motion vector, wherein the fifth image block is a right neighboring image block of the first center image block, the sixth image block is a lower neighboring image block of the first center image block, and the third frame image is a previous frame image adjacent to the first frame image;

[0092] obtaining a zero point image block corresponding to the first center image block in the second frame image, the zero point image block having a same coordinate position in the second frame image as the first center image block has in the first frame image;

[0093] obtaining a global motion vector of the third frame image, the global motion vector being obtained according to clustering of motion vectors corresponding to a plurality of image blocks divided from the third frame image;

[0094] obtaining, according to the global motion vector, an image block corresponding to the first center image block in the second frame image as a global image block;

[0095] the third image block, the fourth image block, the seventh image block, the eighth image block, the zero point image block and the global image block form a plurality of first candidate image blocks in a first candidate search set. Optionally, the coarse search result is a matching result between the first center image block and the plurality of first candidate image blocks, and the second center image block is determined according to the coarse search result, including:

[0096] determining, as the second center image block, an image block having a highest matching degree with the first center image block among the first candidate image blocks.

[0097] Optionally, the processing module is further configured to determine a matching degree between the first center image block and the plurality of first candidate image blocks, and specifically configured to:

[0098] calculating a first absolute error sum of the first center image block and the first candidate image block, the first absolute error sum being a sum of absolute values of differences between pixel values of a plurality of pixel points in the first center image block and a plurality of pixel points to be matched in the first candidate image block;

[0099] summing the first absolute error sum and a first smoothing term to obtain a first estimation value, the first smoothing term being determined according to a difference sum of motion vectors of the first candidate image block and neighboring image blocks of the first center image block;

[0100] determine a matching degree between the first center image block and the first candidate image block according to the first estimation value.

[0101] Optionally, the processing module 302 is further configured to determine a plurality of second candidate image blocks in a second candidate search set, specifically configured to:

[0102] obtain an image block with a first distance from the second center image block as a first step image block;

[0103] determine a third center image block according to a matching result between the second center image block and the first step image block;

[0104] obtain an image block with a first distance from the third center image block as a second step image block;

[0105] obtain an image block with a second distance from the second center image block as a third step image block, the second distance being greater than the first distance;

[0106] determine a fourth center image block according to a matching result between the second center image block and the third step image block;

[0107] obtain an image block with a second distance from the fourth center image block as a fourth step image block;

[0108] The second center image block, the first step image block, the second step image block, the third step image block and the fourth step image block constitute the plurality of second candidate image blocks in the second candidate search set.

[0109] Optionally, the accurate search result is a matching result of the first center image block and the plurality of second candidate image blocks, and a motion estimation result of the first center image block is determined according to the accurate search result, including:

[0110] determine a final matching block according to the accurate search result, the final matching block being an image block with a highest matching degree from the plurality of second candidate image blocks to the first center image block;

[0111] calculate a final motion vector between the first center image block and the final matching block as the motion estimation result of the first center image block.

[0112] Optionally, the processing module 302 is further configured to determine a matching degree between the first center image block and the plurality of second candidate image blocks, specifically configured to:

[0113] calculate a second absolute error sum between the first center image block and the second candidate image block, the second absolute error sum being a sum of absolute values of differences between pixel values of a plurality of pixel points in the first center image block and a plurality of pixel points to be matched in the second candidate image block;

[0114] summing the second absolute error and a second smooth term and a first distance difference, the second smooth term being determined according to a difference between a motion vector of the second candidate image block and motion vectors of neighboring image blocks of the first center image block, the first distance difference being determined according to a maximum value of absolute values of x or y direction offsets of motion vectors between the first center image block and the second candidate image block;

[0115] determining a matching degree between the first center image block and the first candidate image block according to the first estimation value.

[0116] It is worth pointing out that the specific implementation of the motion estimation device can refer to the description of the motion estimation method described above, and will not be described here. Each unit or module in the motion estimation device can be combined into one or several other units or modules, respectively or all, or some of the units or modules can be further divided into a plurality of units or modules with smaller functions to constitute, which can achieve the same operation without affecting the implementation of the technical effects of the embodiments of the present application. The above-mentioned units or modules are divided based on logical functions. In actual application, the function of one unit (or module) can also be realized by multiple units (or modules), or the functions of multiple units (or modules) can be realized by one unit (or module).

[0117] As to each device, product containing module / unit described in the above embodiments, it can be a software module / unit, or a hardware module / unit, or part of it is a software module / unit and part of it is a hardware module / unit. For example, as to each device, product of application or integrated chip, each module / unit contained therein can all be realized by hardware such as circuit, or at least part of the modules / units can be realized by software program, which runs in the integrated processor inside the chip, and the remaining part of the modules / units can be realized by hardware such as circuit; as to each device, product of application or integrated chip module, each module / unit contained therein can all be realized by hardware such as circuit, and different modules / units can be located in the same component (such as chip, circuit module, etc.) or different components of the chip module, at least part of the modules / units can be realized by software program, which runs in the integrated processor inside the chip module, and the remaining part of the modules / units can be realized by hardware such as circuit; as to each device, product of application or integrated terminal, each module / unit contained therein can all be realized by hardware such as circuit, and different modules / units can be located in the same component (such as chip, circuit module, etc.) or different components of the terminal, or at least part of the modules / units can be realized by software program, which runs in the integrated processor inside the terminal, and the remaining part of the modules / units can be realized by hardware such as circuit.

[0118] Based on the descriptions of the method and apparatus embodiments above, this invention also provides a motion estimation device. Please refer to... Figure 4 This is a schematic diagram of the structure of a motion estimation device provided in an embodiment of the present invention. Figure 4 As shown, the motion estimation device 300 described above can be applied to the motion estimation device 400. The motion estimation device 400 may include a processor 401, a network interface 404, and a memory 405. Furthermore, the motion estimation device 400 may also include a user interface 403 and at least one communication bus 402. The communication bus 402 is used to enable communication between these components. The user interface 403 may include a display screen and a keyboard; optionally, the user interface 403 may also include a standard wired interface or a wireless interface. The network interface 404 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 405 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 405 may also be at least one storage device located remotely from the aforementioned processor 401. Figure 4 As shown, the memory 405, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application program.

[0119] exist Figure 4 In the motion estimation device 400 shown, the network interface 404 provides network communication functionality; the user interface 403 is mainly used to provide an input interface for the user; and the processor 401 can be used to call the device control application stored in the memory 405 to implement the steps of the motion estimation method for the video frame image described above.

[0120] It should be understood that the motion estimation device 400 described in the embodiments of the present invention can execute the motion estimation method for video frame images described above, and can also execute the motion estimation device described above, which will not be repeated here. In addition, the beneficial effects of using the same method will not be repeated here either.

[0121] In addition, it is pointed out here that the embodiment of the present application further provides a computer storage medium, and the computer storage medium stores the computer program executed by the video processing device mentioned above, and the computer program includes program instructions, when the processor executes the program instructions, the description of the video processing method described above can be executed, therefore, it will not be described here. In addition, the description of the beneficial effects of using the same method will not be described here. For the technical details of the computer storage medium embodiment of the present application, please refer to the description of the method embodiment of the present application.

[0122] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiment methods. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM), etc.

[0123] The above disclosure is only the preferred embodiment of the present application, and of course cannot limit the scope of the right of the present application, therefore, the equivalent changes made according to the claims of the present application still fall within the scope of the present application.

Claims

1. A motion estimation method for video frame images, characterized in that, The method includes: Acquire the first frame image from the target video and divide the first frame image into multiple image blocks; Obtain the first center image block among the plurality of image blocks; A coarse search is performed on multiple first candidate image blocks in the first candidate search set using the first central image block, and a second central image block is determined based on the coarse search results. The first candidate image blocks are determined based on image blocks in adjacent frame images of the first frame image. Based on the second central image patch, multiple second candidate image patches are determined in the second candidate search set, specifically including: A first-step image block is obtained with a step distance of a first distance from the second center image block; a third center image block is determined based on the matching result between the second center image block and the first-step image block; a second-step image block is obtained with a step distance of the first distance from the third center image block; a third-step image block is obtained with a step distance of the second distance from the second center image block, where the second distance is greater than the first distance; a fourth center image block is determined based on the matching result between the second center image block and the third-step image block; a fourth-step image block is obtained with a step distance of the second distance from the fourth center image block; the second center image block, the first-step image block, the second-step image block, the third-step image block, and the fourth-step image block constitute multiple second candidate image blocks in the second candidate search set; The first central image block is used to perform a precise search on the plurality of second candidate image blocks, and the motion estimation result of the first central image block is determined based on the precise search results; The motion estimation result of the first frame image is determined based on the motion estimation results of multiple image blocks of the first frame image.

2. The method according to claim 1, characterized in that, The precise search result is the matching result between the first central image patch and the plurality of second candidate image patches. Determining the motion estimation result of the first central image patch based on the precise search result includes: The final matching block is determined based on the precise search results. The final matching block is the image block among the plurality of second candidate image blocks that has the highest matching degree with the first central image block. The final motion vector between the first central image block and the final matching block is calculated and used as the motion estimation result of the first central image block.

3. The method according to claim 2, characterized in that, The method further includes determining the matching degree between the first central image patch and the plurality of second candidate image patches, specifically including: The second absolute error sum between the first central image block and the second candidate image block is calculated. The second absolute error sum is the sum of the absolute values ​​of the differences between the pixel values ​​of multiple pixels in the first central image block and multiple pixels to be matched in the second candidate image block. The second absolute error is summed with the second smoothing term and the first distance difference to obtain the second estimate. The second smoothing term is determined based on the difference between the motion vector of the second candidate image block and the motion vector of the adjacent image block of the first center image block. The first distance difference is determined based on the maximum value of the absolute value of the x or y direction offset of the motion vector between the first center image block and the second candidate image block. The matching degree between the first central image block and the second candidate image block is determined based on the second estimate.

4. A motion estimation device, characterized in that, The device includes: The acquisition module is used to acquire the first frame image in the target video; The processing module is used to divide the first frame image into multiple image blocks and obtain the first center image block among the multiple image blocks; The processing module is further configured to perform a coarse search on multiple first candidate image blocks in the first candidate search set using the first central image block, and determine a second central image block based on the coarse search results, wherein the first candidate image block is determined based on image blocks in adjacent frame images of the first frame image; The processing module is further configured to determine multiple second candidate image blocks in the second candidate search set based on the second central image block, specifically including: A first-step image block is obtained with a step distance of a first distance from the second center image block; a third center image block is determined based on the matching result between the second center image block and the first-step image block; a second-step image block is obtained with a step distance of the first distance from the third center image block; a third-step image block is obtained with a step distance of the second distance from the second center image block, where the second distance is greater than the first distance; a fourth center image block is determined based on the matching result between the second center image block and the third-step image block; a fourth-step image block is obtained with a step distance of the second distance from the fourth center image block; the second center image block, the first-step image block, the second-step image block, the third-step image block, and the fourth-step image block constitute multiple second candidate image blocks in the second candidate search set; The processing module is further configured to perform a precise search on the plurality of second candidate image blocks using the first central image block, determine the motion estimation result of the first central image block based on the precise search result, and determine the motion estimation result of the first frame image based on the motion estimation results of the plurality of image blocks of the first frame image.

5. The apparatus according to claim 4, characterized in that, The precise search result is the matching result between the first central image patch and the plurality of second candidate image patches. Determining the motion estimation result of the first central image patch based on the precise search result includes: The final matching block is determined based on the precise search results. The final matching block is the image block among the plurality of second candidate image blocks that has the highest matching degree with the first central image block. The final motion vector between the first central image block and the final matching block is calculated and used as the motion estimation result of the first central image block.

6. The apparatus according to claim 5, characterized in that, The processing module is further configured to determine the matching degree between the first central image block and the plurality of second candidate image blocks, specifically for: The second absolute error sum between the first central image block and the second candidate image block is calculated. The second absolute error sum is the sum of the absolute values ​​of the differences between the pixel values ​​of multiple pixels in the first central image block and multiple pixels to be matched in the second candidate image block. The second absolute error is summed with the second smoothing term and the first distance difference to obtain the second estimate. The second smoothing term is determined based on the difference between the motion vector of the second candidate image block and the motion vector of the adjacent image block of the first center image block. The first distance difference is determined based on the maximum value of the absolute value of the x or y direction offset of the motion vector between the first center image block and the second candidate image block. The matching degree between the first central image block and the second candidate image block is determined based on the second estimate.

7. A motion estimation device, characterized in that, include: Processor and memory; The processor is connected to a memory, wherein the memory is used to store program code, and the processor is used to call the program code to execute the video frame image motion estimation method as described in any one of claims 1-3.

8. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions perform the video frame image motion estimation method as described in any one of claims 1-3.

Images