Motion compensation method, electronic device, and storage medium

Abstract

The application provides a motion compensation method, an electronic device and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: firstly, obtaining a motion vector of a starting search point of a neighboring block of a to-be-processed block in a reference frame and a motion vector of the neighboring block; then, determining a search range in the reference frame according to the motion vector of the starting search point and the motion vector of the neighboring block; and finally, determining a target pixel point in the search range and performing motion compensation on the to-be-processed block based on the motion vector of the target pixel point. In the embodiment, the search range of the to-be-processed block is adaptively determined based on the motion vector of the starting search point of the neighboring block of the to-be-processed block and the motion vector of the neighboring block, so that invalid search can be avoided, the search complexity is reduced, and the search time is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a motion compensation method, an electronic device and a storage medium. BACKGROUND

[0002] Motion compensation is a method for describing the difference between adjacent frames (adjacent here means adjacent in the coding relationship, and the two frames are not necessarily adjacent in the playing order), that is, describing how each block of a reference frame moves to a current frame. Specifically, for each block, find the most similar block (matching block) in a certain search range of the reference frame, output a motion vector (MV), that is, the position coordinates of the matching block relative to the current block, and then realize motion compensation according to the motion vector.

[0003] In the process of searching for a matching block, the search range is generally fixed and unchangeable, which will result in a lot of invalid searches, and the search complexity of this method is high and the search time is long. SUMMARY

[0004] The embodiments of the present application provide a motion compensation method, an electronic device and a storage medium to avoid invalid searches, reduce search complexity and reduce search time.

[0005] In a first aspect, the embodiments of the present application provide a motion compensation method, which comprises:

[0006] obtaining a motion vector of a starting search point of a neighboring block of a to-be-processed block in a reference frame, and a motion vector of the neighboring block in the to-be-processed image;

[0007] determining a search range in the reference frame according to the motion vector of the starting search point and the motion vector of the neighboring block;

[0008] determining a target pixel point in the search range, and performing motion compensation on the to-be-processed block based on a motion vector of the target pixel point.

[0009] In a second aspect, the embodiments of the present application provide an electronic device, which comprises a memory, a processor and a computer program stored in the memory, and the processor realizes the method of any one of the above aspects when executing the computer program.

[0010] In a third aspect, the embodiments of the present application provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the method of any one of the above aspects.

[0011] Compared with the prior art, the present application has the following advantages:

[0012] The application provides a motion compensation method, an electronic device and a storage medium. Firstly, a motion vector of a starting search point of a neighboring block of a to-be-processed block in a reference frame and a motion vector of the neighboring block are acquired; a search range is determined in the reference frame according to the motion vector of the starting search point and the motion vector of the neighboring block; a target pixel point is determined in the search range; and the to-be-processed block is compensated for motion based on the motion vector of the target pixel point. In this embodiment, the search range of the to-be-processed block is adaptively determined based on the motion vector of the starting search point of the neighboring block of the to-be-processed block and the motion vector of the neighboring block, so that invalid search can be avoided, the search complexity is reduced, and the search time is reduced.

[0013] The above description is only a summary of the technical scheme of the application. In order to more clearly understand the technical means of the application, the application can be implemented according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the application more obvious and easy to understand, the following specific embodiments of the application are described. BRIEF DESCRIPTION OF DRAWINGS

[0014] In the drawings, like reference numerals refer to same or similar components throughout the several views, unless otherwise indicated. These drawings are not necessarily to scale. It should be understood that these drawings only depict some embodiments in accordance with the application and should not be considered as limiting the scope of the application.

[0015] Figure 1 An application scenario diagram of the motion compensation method provided by the application;

[0016] Figure 2 A diagram of hierarchical motion estimation for MCTF;

[0017] Figure 3 A flowchart of the motion compensation method of an embodiment of the application;

[0018] Figure 4 A diamond search diagram of an embodiment of the application;

[0019] Figure 5 A diagram of the integer pixel search process of an embodiment of the application;

[0020] Figure 6 A structural block diagram of a motion compensation device of an embodiment of the application; and

[0021] Figure 7 A block diagram of an electronic device for implementing an embodiment of the application. DETAILED DESCRIPTION

[0022] In the following, only certain exemplary embodiments are simply described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the concept or scope of the present application. Therefore, the drawings and the description are considered to be exemplary in nature, rather than limiting.

[0023] To facilitate understanding of the technical solutions of the embodiments of the present application, the related technologies of the embodiments of the present application are described below. The following related technologies can be combined with the technical solutions of the embodiments of the present application in any manner as optional schemes, and all belong to the protection scope of the embodiments of the present application.

[0024] Figure 1 An application scenario of the motion compensation method provided by the present application is shown. The method in the embodiments of the present application can be applied in the multi-functional video coding (VVC) of the video coding standard H.266. Specifically, it can be applied in the motion estimation and motion compensation in the motion compensated temporal filter (MCTF) and inter prediction as shown in Figure 1 The MCTF is applied in the pre-processing of VVC, and is mainly used for filtering the input original video in the time domain through a bilateral filter, which can effectively reduce video noise and improve coding efficiency.

[0025] The coding process of H.266 is shown in Figure 1 Specifically, after the input video is filtered through the MCTF, a filtered video is obtained, intra prediction and inter prediction are performed on the filtered video to obtain prediction residuals, the prediction residuals are subjected to integer transform to change from the spatial domain to the frequency domain, the transformed coefficients are quantized, the quantized coefficients are formed into one-dimensional data through a specific scanning manner, information is extracted therefrom, and then entropy coding is performed to obtain a bit stream which is sent to a user terminal for decoding and playing. In order to provide reference frames for prediction, the encoder needs to reconstruct images, inverse quantization and inverse transform are performed through the inverse operation of coding to obtain residual images, the residual images are added to the prediction values obtained by intra prediction or inter prediction, and then loop filtering is performed to obtain reference frames. Mode decision and encoder logic control are used to determine the mode and logic of coding according to specific needs.

[0026] The MCTF adopts a hierarchical motion estimation scheme as shown in Figure 2 L0 is an image of original resolution, L1 is a down-sampled image of L0, and L2 is a down-sampled image of L1, wherein the width and height of L1 are half of the width and height of L0, and the width and height of L2 are half of the width and height of L1. The calculation process of the MCTF motion estimation is as follows:

[0027] (1) Perform integer pixel motion estimation on each 16x16 block in L3 to get the optimal motion vector MV0;

[0028] (2) Take the motion vector MV0 as the initial value, and then perform the same integer pixel motion estimation on each 16x16 block in L2 to get the optimal motion vector MV1;

[0029] (3) Take the motion vector MV1 as the initial value, and then perform the same integer pixel motion estimation on each 16x16 block in L1 to get the optimal motion vector MV2;

[0030] (4) Take the motion vector MV2 as the initial value, and then perform integer pixel motion estimation and fractional pixel motion estimation on each 8x8 block in L0 to get the final motion vector MV3. Motion compensation and bilateral filtering are performed using the motion vector MV3.

[0031] In the integer pixel motion estimation and fractional pixel motion estimation in each layer, the optimal integer pixel is selected by searching for integer pixels within a certain range centered on the starting search point, and the matching block is determined based on the search results of the fractional pixels searched around the optimal integer pixel. The search range is fixed, and all integer pixels are searched in the fixed search range, and then the fractional pixels are searched based on the search results of the integer pixels, which results in a lot of invalid searches and high search complexity.

[0032] To solve the above problems, in a related technology, all integer pixels within a certain range and all fractional pixels within a range of one integer pixel around the optimal integer pixel are sequentially traversed, or the search range of the integer pixels is directly reduced and the search of the fractional pixels is directly reduced by a certain step size. Although this scheme can reduce the complexity to a certain extent, it has a large loss in performance (e.g., peak signal-to-noise ratio, structural similarity), which affects the encoding quality.

[0033] In another related technology, the search is performed by step size from large to small. This scheme can reduce the complexity to a certain extent, but for the hierarchical search of MCTF, most step sizes will not be large. Such a search method will waste part of the search time and cannot maximize the reduction of the search time of the integer pixels and fractional pixels of MCTF.

[0034] Therefore, the embodiment of the present application provides a motion compensation method. Firstly, a motion vector of a starting search point of a neighboring block of a to-be-processed block in a reference frame and a motion vector of the neighboring block are obtained. Then, a search range of the to-be-processed block is determined in the reference frame according to the motion vector of the starting search point and the motion vector of the neighboring block. A target pixel point is determined in the search range, and the to-be-processed block is compensated for motion based on the motion vector of the target pixel point. In the embodiment, the search range of the to-be-processed block is adaptively determined based on the motion vector of the starting search point of the neighboring block and the motion vector of the neighboring block, so that invalid search can be avoided, the search complexity is reduced, and the search time is reduced.

[0035] As shown in FIG. 1, the motion compensation method includes the following steps. Figure 3 As shown in FIG. 1, the motion compensation method includes the following steps.

[0036] In step S301, a motion vector of a starting search point of a neighboring block of a to-be-processed block in a reference frame and a motion vector of the neighboring block are obtained.

[0037] In step S302, a search range of the to-be-processed block is determined in the reference frame according to the motion vector of the starting search point and the motion vector of the neighboring block.

[0038] In step S303, a target pixel point is determined in the search range, and the to-be-processed block is compensated for motion based on the motion vector of the target pixel point.

[0039] The method in the embodiment can be applied in a computing device, which can include a server, a user terminal, and the like. The to-be-processed image can be any one of multiple layers of images obtained by down-sampling an original image frame. The starting search point of the to-be-processed block is determined in the reference frame according to an optimal MV of the to-be-processed block at a corresponding position of a previous layer of image and blocks around the corresponding position (if there is no previous layer of image, a preset optimal MV value, for example, zero).

[0040] For the neighboring block of the to-be-processed block for which an optimal pixel point has been searched, the motion vector of the starting search point of the neighboring block and the motion vector of the neighboring block are obtained. The motion vector of the neighboring block can be a motion vector of a target pixel point corresponding to the neighboring block, the target pixel point corresponding to the neighboring block is a pixel point corresponding to a matching block of the neighboring block, and the matching block of the neighboring block is a pixel block in the search range of the neighboring block and between the neighboring block, for which a matching cost value is the smallest.

[0041] According to the motion vector of the starting search point of the neighboring block and the motion vector of the neighboring block, a search range of the to-be-processed block is determined, in the search range of the to-be-processed block, a target pixel point corresponding to the to-be-processed block is searched starting from the starting search point of the to-be-processed block, the target pixel point corresponding to the to-be-processed block is a pixel point corresponding to a matching block of the to-be-processed block, for example, the target pixel point can be a pixel point at the upper left corner of the matching block, and the matching block of the to-be-processed block can be determined according to the target pixel point. The target pixel point can be an integer pixel point or a fractional pixel point.

[0042] The neighboring block can include at least one block adjacent to the to-be-processed block above, below, left or right.

[0043] Based on the optimal motion vector corresponding to the to-be-processed block, that is, the motion vector of the target pixel point searched in the search range, a matching block of the to-be-processed block is determined in the reference frame, and the to-be-processed block is motion compensated.

[0044] The motion compensation method provided in the embodiments of the present application first acquires the motion vector of the starting search point of the neighboring block of the to-be-processed block in the reference frame and the motion vector of the neighboring block, then determines a search range in the reference frame according to the motion vector of the starting search point and the motion vector of the neighboring block, determines a target pixel point in the search range, and motion compensates the to-be-processed block based on the motion vector of the target pixel point. In the embodiments, the search range of the to-be-processed block is adaptively determined based on the motion vector of the starting search point of the neighboring block of the to-be-processed block and the motion vector of the neighboring block, which can avoid invalid search, reduce search complexity, and reduce search time.

[0045] Specifically, how to determine the search range is shown in the following embodiments:

[0046] In one implementation, the search range in the reference frame is determined according to the motion vector of the starting search point and the motion vector of the neighboring block, including: determining a search edge length based on the distance between the motion vector of the starting search point of the neighboring block and the motion vector of the neighboring block; and determining the search range in the reference frame according to the search edge length of the search range.

[0047] In the embodiments, the distance between the motion vector of the starting search point of the neighboring block and the motion vector of the neighboring block is acquired, the search edge length is determined according to the distance, and then the search range is determined. Since the neighboring blocks have correlation, the search edge length of the search range of the current block is adaptively determined according to the motion vector of the starting search point of the neighboring block and the motion vector of the neighboring block, which can avoid a lot of invalid search, reduce the complexity of pixel search, and guarantee rate-distortion performance.

[0048] In an implementation, the number of the adjacent blocks is one, and the search side length of the search range is determined based on the distance between the motion vector of the starting search point of the adjacent block and the motion vector of the adjacent block, including: taking the distance between the motion vector of the starting search point of the adjacent block and the motion vector of the adjacent block as the search parameter value of the adjacent block; and determining the search side length of the search range according to the search parameter value of the adjacent block.

[0049] In actual application, the distance between the motion vector of the starting search point of one adjacent block and the motion vector of the adjacent block can be taken as the search parameter value of the adjacent block, the search side length of the search range of the to-be-processed block is determined according to the search parameter value, and the search range of the to-be-processed block is determined according to the search side length.

[0050] In an example, the search side length is determined according to the following formula:

[0051] cur_range = min(prev_range + x, a) (1)

[0052] Wherein, cur_range represents the search side length of the to-be-processed block, prev_range represents the search parameter value of the adjacent block on the left of the to-be-processed block, the search parameter value of the adjacent block is the distance between the motion vector of the starting search point of the adjacent block and the motion vector of the adjacent block. x is a preset redundancy threshold, and a is a fixed parameter value, which can be set according to specific needs, and x can be any value in [0, a]. In the embodiment, a = 16, compared with the fixed search range side length of 16 in the related art, the search range is reduced, and invalid search is avoided to a certain extent. In addition, the distance between the motion vector of the starting search point of the adjacent block and the motion vector of the adjacent block can be multiplied by the redundancy threshold to obtain the parameter value of the search range of the to-be-processed block, and the search range of the to-be-processed block is further determined.

[0053] In an implementation, the number of the adjacent blocks is multiple, and the search side length of the search range is determined based on the distance between the motion vector of the starting search point of the adjacent block and the motion vector of the adjacent block, including: obtaining the distance between the motion vector of the starting search point of each of the multiple adjacent blocks and the motion vector of the adjacent block, performing weighted calculation on the multiple distances to obtain a search parameter value; and determining the search side length of the search range according to the search parameter value.

[0054] In actual application, the distance between the motion vector of the starting search point of each of the multiple adjacent blocks and the motion vector of the adjacent block can be calculated, the multiple distances can be weighted calculated to obtain a search parameter value, the search side length of the search range of the to-be-processed block is determined according to the search parameter value, and the search range of the to-be-processed block is determined according to the search side length.

[0055] For example, the search parameter value prev_range in formula (1) can be calculated by weighting distances corresponding to a plurality of adjacent blocks.

[0056] After the search edge length of the search range of the to-be-processed block is determined, a square region obtained by taking the starting search point of the to-be-processed block as a center and taking the search edge length as an edge length is the search range of the to-be-processed block. For example, the search edge length of the search range of the to-be-processed block is 5, and a 5-pixel-point-by-5-pixel-point square region with the starting search point of the to-be-processed block as a center is the search range of the to-be-processed block. In this way, as the search parameter values of the adjacent blocks change continuously, the search range of the to-be-processed block dynamically changes in size. Compared with the manner of using a fixed-size search range in the related art, the manner can avoid invalid search, reduce search complexity, and reduce search time.

[0057] In an implementation manner, in the search range, the target pixel point is determined, including: searching for an optimal integer pixel point in the search range in a diamond search manner starting from the starting search point of the to-be-processed block; searching for an optimal fractional pixel point in the search range in the diamond search manner taking the searched optimal integer pixel point as a diamond center point; and determining the target pixel point from the optimal integer pixel point and the optimal fractional pixel point.

[0058] In actual application, first, an optimal integer pixel point is searched for in the search range in a diamond search manner starting from the starting search point of the to-be-processed block, an optimal fractional pixel point is searched for taking the optimal integer pixel point as a diamond center point, and a target pixel point is selected from the optimal integer pixel point and the optimal fractional pixel point, which has the minimum matching cost value between a pixel block corresponding to the pixel point and the to-be-processed block. In this embodiment, compared with sequentially searching for all pixel points in a fixed search range in the related art, the integer pixel point and the fractional pixel point are searched for in the diamond search manner, which can greatly reduce search complexity while ensuring performance.

[0059] In an implementation manner, in the search range, the optimal integer pixel point is searched for in the diamond search manner starting from the starting search point of the to-be-processed block, including: taking the starting search point as a diamond center point, determining matching cost values between pixel blocks corresponding to integer pixel points adjacent to the diamond center point and the to-be-processed block in an upper direction, a lower direction, a left direction, and a right direction of the diamond center point, respectively, taking an integer pixel point corresponding to a pixel block having the minimum matching cost value between the pixel block and the to-be-processed block as an optimal integer pixel point of a current round of diamond search, and taking the optimal integer pixel point of the current round of diamond search as a diamond center point of a next round of diamond search to search for an optimal integer pixel point of the next round.

[0060] The matching cost value for each point is calculated based on the pixel values ​​of each pixel in the pixel block with that point as the top-left corner in the reference frame and the pixel values ​​of each pixel in the block to be processed. The matching cost value measures the similarity between pixels; the higher the matching cost value, the less similar the corresponding pixels in the two images are.

[0061] In the first round of the diamond search, the starting search point is used as the center point of the diamond. The matching cost between the pixel block corresponding to the center point and its adjacent integer pixels above, below, to the left, and to the right of the center point and the block to be processed is calculated. The integer pixel corresponding to the pixel block with the smallest matching cost is taken as the optimal integer pixel of the current round of the diamond search. In subsequent search rounds, the optimal integer pixel found in the previous search round can be used as the center point of the diamond to determine a new diamond and continue the diamond search.

[0062] In one example, such as Figure 4 As shown, the points searched in this example are integer pixels. Each triangle in the figure represents an integer pixel. In this embodiment, the search range is a 5×5 area. If it is the first round of search, the starting search point a is taken as the center point of the rhombus. The matching cost value corresponding to the center point a and the integer pixels e, d, b, and c above, below, to the left, and to the right of the center point a is calculated respectively. The matching cost value corresponding to these 5 points is compared, and the point with the smallest matching cost value is taken as the optimal integer pixel point of the current round of rhombus search. If the optimal integer pixel point of the current round of rhombus search is point c, then the next round of rhombus search uses point c as the center point of the rhombus and searches for points a, c, h, g, and f. By calculating the matching cost value, g is determined to be the optimal integer pixel point. Then, in the next round, point g is taken as the center point of the rhombus and points d, g, i, c, and j are calculated, and so on, until the final optimal integer pixel point is determined in the search range.

[0063] In one implementation, within the search range, starting from the initial search point of the block to be processed, the optimal integer pixel is searched using a diamond search method. This includes: within the search range, starting from the initial search point of the block to be processed, determining the removal point for the next round of diamond search based on the matching cost between the pixel blocks corresponding to the multiple integer pixels in the current round of diamond search and the block to be processed; and in the next round of diamond search, searching for the optimal integer pixel among the integer pixels other than the removal point.

[0064] In practical applications, it is not necessary to search every pixel in the search range. The optimal integer pixel for the current round of diamond search can be determined based on the matching cost of each integer pixel in the current round. Based on the position of the optimal integer pixel, the removal point for the next round, i.e. the point that does not need to be searched, can be determined, thereby reducing the search time.

[0065] Specifically how to determine the removal point of the next round, see the following embodiments:

[0066] In an implementation, from the starting search point of the to-be-processed block in the search range, the removal point of the next round of diamond search is determined according to the matching cost values between the pixel blocks corresponding to the integral pixel points of the current round of diamond search and the to-be-processed block, comprising:

[0067] In the search range, from the starting search point of the to-be-processed block, the optimal integral pixel point of the current round of diamond search is determined according to the matching cost values between the pixel blocks corresponding to the integral pixel points of the current round of diamond search and the to-be-processed block; if the optimal integral pixel point of the current round of diamond search is the integral pixel point adjacent to the left or right of the diamond center point, the removal point of the next round of diamond search is determined as the integral pixel point with a larger matching cost value among the integral pixel points adjacent to the top and bottom of the diamond center point; if the optimal integral pixel point of the current round of diamond search is the integral pixel point adjacent to the top or bottom of the diamond center point, the removal point of the next round of diamond search is determined as the integral pixel point with a larger matching cost value among the integral pixel points adjacent to the left and right of the diamond center point.

[0068] In actual application, from the starting search point of the to-be-processed block, the matching cost values corresponding to the diamond center point, the points adjacent to the left, right, top and bottom of the diamond center point are calculated, and the point with the smallest matching cost value is taken as the optimal integral pixel point. If the optimal integral pixel point is the integral pixel point adjacent to the left or right of the diamond center point, the matching cost values corresponding to the integral pixel points adjacent to the top and bottom of the diamond center point are compared, and the position of the removal point in the next round of search is determined according to the position of the integral pixel point with a larger matching cost value. For example, if the matching cost value of the integral pixel point adjacent to the top is larger, the integral pixel point adjacent to the top of the diamond center point is taken as the removal point in the next round of diamond search, and the matching cost value of the point is not calculated, and the matching cost values of the other points in the diamond are calculated, thereby reducing the calculation time.

[0069] If the optimal integral pixel point is the integral pixel point adjacent to the top or bottom of the diamond center point, the matching cost values corresponding to the integral pixel points adjacent to the left and right of the diamond center point are compared, and the position of the removal point in the next round of search is determined according to the position of the integral pixel point with a larger matching cost value. For example, if the matching cost value of the integral pixel point adjacent to the left is larger, the integral pixel point adjacent to the left of the diamond center point is taken as the removal point in the next round of diamond search, and the matching cost value of the point is not calculated, and the matching cost values of the other points in the diamond are calculated, thereby reducing the calculation time.

[0070] In the optimal pixel point search process, how to stop the search, see the following embodiments:

[0071] In an implementation, the method further comprises: if the diamond center point of the current round of diamond search is the optimal integer pixel point, taking the current obtained optimal integer pixel point as the optimal integer pixel point of the block to be processed.

[0072] In actual application, if the diamond center point of the current round of diamond search is the optimal integer pixel point, i.e., the matching cost value of the diamond center point is the minimum, the search is not continued, and the optimal integer pixel point of the block to be processed is obtained. Similarly, when the optimal fractional pixel point is obtained, if the diamond center point of the current round of diamond search is the optimal fractional pixel point, i.e., the matching cost value of the diamond center point is the minimum, the search is not continued, and the optimal fractional pixel point of the block to be processed is obtained. Then, the matching cost values corresponding to the optimal integer pixel point and the optimal fractional pixel point are compared, and the one with the minimum matching cost value is taken as the target pixel point.

[0073] In an implementation, the method further comprises: if the closeness of the maximum value and the minimum value of the matching cost value of the current round of diamond search meets a preset condition, and the number of times of stopping search exceeds a preset threshold, taking the current obtained optimal integer pixel point as the optimal integer pixel point of the block to be processed.

[0074] wherein the initial value of the number of times of stopping search is a preset value, and the number of times of stopping search is increased until the closeness of the maximum value and the minimum value of the matching cost value of the current round of diamond search meets the preset condition, and the number of times of stopping search exceeds the preset threshold, then the integer pixel point search of the next round is stopped.

[0075] wherein the preset condition can be min_value*x>max_value, wherein x is a preset value, and can be set according to specific needs.

[0076] In the following, the integer pixel point search process of the technical scheme of the present application is introduced through a specific embodiment. In combination with Figure 4 and Figure 5 , the integer pixel point search method provided by the embodiment of the present application comprises:

[0077] 1) first setting the number of times of stopping search stop_count as 0, and setting the search round number search_count as 0, and taking the point a in Figure 4 as the diamond center point of the first round;

[0078] 2) judging whether the value of search_count is greater than T1, wherein T1 is a preset value, and the value is greater than or equal to 0; if yes, stopping the integer pixel search, and taking the optimal point of the last round as the optimal point of the final integer pixel search;

[0079] If no, judge whether search_count is 0, if yes, search the up, down, left and right four integer pixel points of the center point of the current diamond (if it is not the first round of search, the removed point does not need to be searched), and calculate the matching cost value cost of each integer pixel point.

[0080] 3) Judge whether the cost of the center point of the current diamond is the minimum, if yes, exit the integer pixel search process, and set the center point of the diamond as the optimal integer pixel point; if no, enter the next step.

[0081] 4) Judge the maximum value max_value and the minimum value min_value of the cost of the center point of the diamond and the up, down, left and right four integer pixel points, if min_value*x is greater than max_value, add 1 to the value of stop_count, otherwise, change the value of stop_count to 0. The value of x is a preset value, which can be any value in [1, 2], including decimal; after multiplying min_value by x, the size comparison with max_value is performed, the purpose is to judge whether min_value and max_value are close;

[0082] 5) Judge whether stop_count is greater than T2, wherein T2 is a preset value, and the value is greater than 0; if yes, stop the integer pixel search, set the optimal integer pixel point position, set the point with the minimum cost in the four integer pixel points of the current diamond as the optimal point of the integer pixel search, and end the integer pixel point search; if no, enter the next step.

[0083] 6) Set the diamond removed point, if the optimal point of the current diamond search Figure 4 (a, b, c, d, e) is the left or right position in the diamond Figure 4 (b point or c point), judge the cost size relationship of the up and down points Figure 4 (d point and e point), if the cost of the upper edge point e is greater than the cost of the lower edge point d, set the upper edge point of the next round of diamond search (if the optimal point of the current search is c point, the next round of search points are a, c, h, f, g, and the upper edge point is f point) as the removed point, otherwise, set the lower edge point Figure 4 (g point) as the removed point; if the optimal point of the current search is the up and down position in the diamond, the same method is used for removal. Then set the optimal point of the current search as the center point of the next round of diamond search, add 1 to the value of search_count, re-enter step 2, continue the search, until the end condition is met, and the optimal integer pixel point is obtained.

[0084] In searching the sub-pixel, the same search method can be adopted, and the pixel with a smaller matching cost in the optimal integer pixel and the optimal sub-pixel is taken as the optimal pixel, and the motion vector corresponding to the optimal pixel is taken as the optimal motion vector.

[0085] Corresponding to the application scenario and method of the method provided by the embodiments of the present application, the embodiments of the present application further provide a motion compensation device. As shown in Figure 6 FIG. 1 is a structural block diagram of a motion compensation device according to an embodiment of the present application. The device includes:

[0086] The acquisition module 601 is configured to acquire a motion vector of a starting search point of a neighboring block of a to-be-processed block in a reference frame and a motion vector of the neighboring block in the to-be-processed image.

[0087] The determination module 602 is configured to determine a search range in the reference frame according to the motion vector of the starting search point and the motion vector of the neighboring block.

[0088] The compensation module 603 is configured to determine a target pixel in the search range, and perform motion compensation on the to-be-processed block based on a motion vector of the target pixel.

[0089] The motion compensation device provided by the embodiments of the present application first acquires a motion vector of a starting search point of a neighboring block of a to-be-processed block in a reference frame and a motion vector of the neighboring block in the to-be-processed image, then determines a search range in the reference frame according to the motion vector of the starting search point and the motion vector of the neighboring block, and finally determines a target pixel in the search range, and performs motion compensation on the to-be-processed block based on a motion vector of the target pixel. In this embodiment, the search range of the to-be-processed block is adaptively determined based on the motion vector of the starting search point of the neighboring block of the to-be-processed block and the motion vector of the neighboring block, which can avoid invalid search, reduce search complexity, and reduce search time.

[0090] In an implementation manner, the determination module 602 is configured to:

[0091] determine a parameter value of the search range based on a distance between the motion vector of the starting search point of the neighboring block and the motion vector of the neighboring block, and determine the search range in the reference frame according to a search side length of the search range.

[0092] In an implementation manner, the number of the neighboring blocks is 1, and when the determination module 602 determines the search side length of the search range based on the distance between the motion vector of the starting search point of the neighboring block and the motion vector of the neighboring block, the determination module 602 is configured to:

[0093] The distance between the motion vector of the starting search point of the neighboring block and the motion vector of the neighboring block is obtained as a search parameter value of the neighboring block; and the search edge length of the search range is determined according to the search parameter value of the neighboring block.

[0094] In an implementation manner, the determining module 602 is configured to: obtain the distance between the motion vector of the starting search point of the neighboring block and the motion vector of the neighboring block, and determine the search edge length of the search range according to the distance.

[0095] The distance between the motion vector of the starting search point of the neighboring block and the motion vector of the neighboring block is obtained as a search parameter value of the neighboring block; and the search edge length of the search range is determined according to the search parameter value of the neighboring block.

[0096] In an implementation manner, the determining module 602 is configured to: obtain the distance between the motion vector of the starting search point of the neighboring block and the motion vector of the neighboring block, and determine the search edge length of the search range according to the distance.

[0097] In the search range, the optimal integer pixel point is searched in a diamond search manner from the starting search point of the to-be-processed block.

[0098] The optimal integer pixel point searched in the diamond search manner is taken as a diamond center point, and the optimal fractional pixel point is searched in a diamond search manner.

[0099] The optimal integer pixel point searched in the diamond search manner is taken as a diamond center point, and the optimal fractional pixel point is searched in a diamond search manner.

[0100] In an implementation manner, the determining module 602 is configured to: obtain the distance between the motion vector of the starting search point of the neighboring block and the motion vector of the neighboring block, and determine the search edge length of the search range according to the distance.

[0101] The starting search point of the to-be-processed block is taken as a diamond center point, the matching cost values between the pixel blocks corresponding to the integer pixel points adjacent to the diamond center point and the to-be-processed block are determined, and the integer pixel point corresponding to the pixel block with the minimum matching cost value is taken as the optimal integer pixel point of the current round of diamond search.

[0102] The optimal integer pixel point of the current round of diamond search is taken as the diamond center point of the next round of diamond search, and the optimal integer pixel point of the next round is searched.

[0103] In an implementation manner, the determining module 602 is configured to: obtain the distance between the motion vector of the starting search point of the neighboring block and the motion vector of the neighboring block, and determine the search edge length of the search range according to the distance.

[0104] In the search range, the optimal integer pixel point is searched in a diamond search manner from the starting search point of the to-be-processed block.

[0105] In the next round of diamond search, the optimal integer pixel point is searched in the integer pixel points other than the removed point.

[0106] In an implementation manner, when the compensation module 603 determines the removed point of the next round of diamond search according to the matching cost values between the pixel blocks corresponding to the integer pixel points of the current round of diamond search and the to-be-processed block, from the starting search point of the to-be-processed block in the search range, the compensation module 603 is configured to:

[0107] In the search range, the optimal integer pixel point of the current round of diamond search is determined according to the matching cost values between the pixel blocks corresponding to the integer pixel points of the current round of diamond search and the to-be-processed block, from the starting search point of the to-be-processed block; if the optimal integer pixel point of the current round of diamond search is the integer pixel point adjacent to the left or right of the diamond center point, the removed point of the next round of diamond search is determined as the integer pixel point with a larger matching cost value among the integer pixel points adjacent to the top and bottom of the diamond center point.

[0108] If the optimal integer pixel point of the current round of diamond search is the integer pixel point adjacent to the top or bottom of the diamond center point, the removed point of the next round of diamond search is determined as the integer pixel point with a larger matching cost value among the integer pixel points adjacent to the left and right of the diamond center point.

[0109] In an implementation manner, the apparatus is further configured to:

[0110] If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the to-be-processed block.

[0111] In an implementation manner, the apparatus is further configured to:

[0112] If the closeness of the maximum value and the minimum value of the matching cost values of the current round of diamond search meets the preset condition, and the number of times of stopping search exceeds the preset threshold, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the to-be-processed block.

[0113] The functions of each module in each apparatus in the embodiments of the present application can be referred to the corresponding description in the above method, and have the corresponding beneficial effects, which will not be described here again.

[0114] Figure 7 A block diagram of an electronic device configured to implement embodiments of the present application is shown. As shown in the figure, the electronic device includes a memory 710 and a processor 720, and the memory 710 stores a computer program capable of running on the processor 720. The processor 720 implements the method in the above embodiments when executing the computer program. The number of the memory 710 and the processor 720 can be one or more. Figure 7

[0115] ​The electronic device further includes:

[0116] The communication interface 730 is configured to communicate with external devices and transmit data.

[0117] If the memory 710, the processor 720 and the communication interface 730 are implemented independently, the memory 710, the processor 720 and the communication interface 730 can be connected to each other through a bus and complete communication therebetween. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0118] Optionally, in a specific implementation, if the memory 710, the processor 720 and the communication interface 730 are integrated on a chip, the memory 710, the processor 720 and the communication interface 730 can complete communication therebetween through an internal interface.

[0119] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method provided in the embodiment of the present application.

[0120] The embodiment of the present application further provides a chip, which includes a processor, is configured to call and run instructions stored in a memory, and makes a communication device installed with the chip execute the method provided in the embodiment of the present application.

[0121] The embodiment of the present application further provides a chip, which includes an input interface, an output interface, a processor and a memory, the input interface, the output interface, the processor and the memory are connected through an internal connection path, and the processor is configured to execute code in the memory, and when the code is executed, the processor is configured to execute the method provided in the embodiment of the present application.

[0122] It is to be understood that the above-mentioned processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. It is to be noted that the processor can be a processor supporting an advanced RISC machine (ARM) architecture.

[0123] Further, the memory can include a read-only memory and a random access memory, optionally. The memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory, for example. The volatile memory can include a random access memory (RAM) used as an external cache. Many forms of RAM are available by way of example but not limitation. For example, a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous DRAM (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a SyncLink DRAM (SLDRAM), and a direct Rambus RAM (DR RAM) can be used.

[0124] In the above-described embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded on a computer, all or part of the processes or functions according to the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium.

[0125] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, a person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0126] In addition, the terms "first", "second", etc. are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0127] Any process or method described in the flowchart or otherwise described herein can be understood as a representation of code including one or more executable instructions for performing a specific logical function or process. Also, the scope of the preferred embodiments of the present application includes additional implementations that can not be shown or discussed explicitly, including implementations that perform functions in a different order, in substantially simultaneous fashion, or in reverse order according to the functions involved.

[0128] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing the logic function, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus, such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions, or in conjunction with these instructions execution system, device or apparatus.

[0129] It should be understood that each part of the present application can be realized by hardware, software, firmware or a combination thereof. In the above embodiments, a plurality of steps or methods can be realized by software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above-mentioned embodiment methods can be completed by a program instructing the relevant hardware, which can be stored in a computer readable storage medium and includes one or a combination of the steps of the embodiment methods when executed.

[0130] In addition, each functional unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software functional module. The above-mentioned integrated module, if realized in the form of a software functional module and sold or used as an independent product, can also be stored in a computer readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0131] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of motion compensation, characterized by, The method comprises: obtaining a motion vector of a starting search point of a neighboring block of a to-be-processed block in a to-be-processed image in a reference frame, and a motion vector of the neighboring block; determining a search range in the reference frame according to the motion vector of the starting search point and the motion vector of the neighboring block; in the search range, starting from the starting search point of the to-be-processed block, searching for an optimal integer pixel point in a diamond search manner, taking the optimal integer pixel point as a diamond center point, searching for an optimal fractional pixel point in a diamond search manner, and determining a target pixel point from the optimal integer pixel point and the optimal fractional pixel point, and performing motion compensation on the to-be-processed block based on a motion vector of the target pixel point.

2. The method of claim 1, wherein, The method comprises: determining a search edge length of the search range based on a distance between the motion vector of the starting search point of the neighboring block and the motion vector of the neighboring block; determining the search range in the reference frame according to the search edge length.

3. The method of claim 2, wherein, The number of neighboring blocks is one, and the method comprises: taking the distance between the motion vector of the starting search point of the neighboring block and the motion vector of the neighboring block as a search parameter value of the neighboring block; determining the search edge length of the search range according to the search parameter value of the neighboring block.

4. The method of claim 2, wherein, The number of neighboring blocks is multiple, and the method comprises: obtaining distances between motion vectors of starting search points corresponding to the multiple neighboring blocks respectively and the motion vectors of the multiple neighboring blocks, performing weighted calculation on the multiple distances to obtain a search parameter value, and determining the search edge length of the search range according to the search parameter value.

5. The method of claim 1, wherein, The method comprises: taking the starting search point of the to-be-processed block as the diamond center point, determining matching cost values between pixel blocks corresponding to integer pixel points adjacent to the diamond center point in an upper direction, a lower direction, a left direction and a right direction of the diamond center point and the to-be-processed block respectively, and taking an integer pixel point corresponding to a pixel block with the smallest matching cost value between the to-be-processed block as an optimal integer pixel point of current round diamond search; taking the optimal integer pixel point of the current round diamond search as a diamond center point of next round diamond search, and searching for an optimal integer pixel point of the next round.

6. The method of claim 1, wherein, The method comprises: in the search range, starting from the starting search point of the to-be-processed block, determining a removal point of next round diamond search according to matching cost values between pixel blocks corresponding to multiple integer pixel points of current round diamond search and the to-be-processed block respectively. In the next round of diamond search, the optimal integer pixel point is searched among the integer pixel points other than the removal point.

7. The method of claim 6, wherein, The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed.

8. The method according to any one of claims 1 to 7, characterized in that, The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed.

9. The method according to any one of claims 1 to 7, characterized in that, The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed.

10. An electronic device, comprising: The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be processed. The method further comprises: If the diamond center point of the current round of diamond search is the optimal integer pixel point, the optimal integer pixel point obtained currently is taken as the optimal integer pixel point of the block to be

Images