Motion estimation method, apparatus, computing device, and computer program product

Abstract

Embodiments of the present application provide a motion estimation method, device, computing equipment and computer program product. The motion estimation method comprises: obtaining a to-be-processed image block in a current image; performing motion estimation on the to-be-processed image block by using a first algorithm to obtain a first motion vector corresponding to the to-be-processed image block; determining a plurality of candidate search centers according to the first motion vector and a second motion vector, wherein the second motion vector is a motion vector corresponding to at least one associated image block having an association relationship with the to-be-processed image block; and performing motion estimation on the to-be-processed image block by using a second algorithm based on the plurality of candidate search centers to obtain a third motion vector corresponding to the to-be-processed image block, wherein the estimation accuracy of the second algorithm is higher than that of the first algorithm, thereby solving the technical problem of low estimation efficiency of the motion estimation method in the related art.

Description

Technical Field

[0001] This application relates to the field of video coding technology, and in particular to a motion estimation method, motion estimation device, computing device, and computer program product. Background Technology

[0002] In video coding, motion estimation techniques are typically used to determine the motion position of the current image patch within a reference image, thereby reducing temporal redundancy and improving video compression efficiency. Existing video coding standards usually employ a hierarchical motion estimation structure, performing multi-stage motion searches on image patches of different sizes to progressively improve motion estimation accuracy.

[0003] For example, the encoder can first obtain an initial motion vector using a low-precision motion estimation method, and then perform a higher-precision motion search based on the initial motion vector to obtain a more accurate motion estimation result. In the above process, corresponding motion vectors can be generated for image blocks of different sizes, and subsequent motion search processes can be performed separately for each block.

[0004] However, as video resolution and coding complexity continue to increase, the number of searches and matching calculations in the motion estimation process continue to increase, resulting in the motion estimation process consuming more hardware resources and computing time, thus affecting video coding efficiency. Summary of the Invention

[0005] This application provides a motion estimation method, a motion estimation device, a computing device, and a computer program product to solve the technical problem of low estimation efficiency in related technologies.

[0006] In a first aspect, embodiments of this application provide a motion estimation method, comprising: acquiring an image patch to be processed in a current image; performing motion estimation on the image patch to be processed using a first algorithm to obtain a first motion vector corresponding to the image patch to be processed; determining a plurality of candidate search centers based on the first motion vector and a second motion vector, wherein the second motion vector is a motion vector corresponding to at least one associated image patch that has an association relationship with the image patch to be processed; and performing motion estimation on the image patch to be processed using a second algorithm based on the plurality of candidate search centers to obtain a third motion vector corresponding to the image patch to be processed, wherein the estimation accuracy of the second algorithm is higher than the estimation accuracy of the first algorithm.

[0007] Secondly, embodiments of this application provide a motion estimation device, comprising: an acquisition module for acquiring an image patch to be processed in a current image; a first estimation module for performing motion estimation on the image patch to be processed using a first algorithm to obtain a first motion vector corresponding to the image patch to be processed; a determination module for determining a plurality of candidate search centers based on the first motion vector and a second motion vector, wherein the second motion vector is a motion vector corresponding to at least one associated image patch that has an association relationship with the image patch to be processed; and a second estimation module for performing motion estimation on the image patch to be processed using a second algorithm based on the plurality of candidate search centers to obtain a third motion vector corresponding to the image patch to be processed, wherein the estimation accuracy of the second algorithm is higher than the estimation accuracy of the first algorithm; wherein the hardware components corresponding to the acquisition module, the first estimation module, the determination module, and the second estimation module are independent of each other and are used to process the motion estimation of the image patch to be processed based on different candidate search centers in parallel; wherein the reference pixel data acquired by the acquisition module is reused for the matching cost calculation of the plurality of candidate search centers.

[0008] Thirdly, embodiments of this application provide a computing device, including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement the motion estimation method provided in the first aspect above.

[0009] Fourthly, embodiments of this application provide a computer program product, including a computer program / instruction, which, when executed by a processing component, implements the motion estimation method provided in the first aspect above.

[0010] In this embodiment, an image block to be processed in the current image is obtained; a first algorithm is used to estimate the motion of the image block to be processed to obtain a first motion vector corresponding to the image block to be processed; based on the first motion vector and the second motion vector, multiple candidate search centers are determined, wherein the second motion vector is the motion vector corresponding to at least one associated image block that has an association with the image block to be processed; based on the multiple candidate search centers, a second algorithm is used to estimate the motion of the image block to be processed to obtain a third motion vector corresponding to the image block to be processed, wherein the estimation accuracy of the second algorithm is higher than that of the first algorithm. That is, a motion estimation algorithm with lower precision is first used to estimate the motion of the image block to be processed, and the first motion vector corresponding to the image block to be processed is obtained. Based on the first motion vector, the motion vectors corresponding to at least one associated image block associated with the image block to be processed can be combined, and a higher precision motion estimation algorithm can be used to estimate the motion of the image block to be processed. Since the associated image block is related to the image block to be processed, the precision of the motion estimation method can be improved. Moreover, the associated image block can be the image block whose corresponding motion vector has been calculated in the lower precision motion estimation algorithm. Therefore, the motion vector corresponding to the associated image block can be obtained more conveniently. This solves the technical problem of low estimation efficiency of motion estimation methods in related technologies and achieves the beneficial effect of improving estimation efficiency. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart of a motion estimation method provided in this application is shown; Figure 2 This is a schematic diagram of a specific embodiment of motion estimation using a hierarchical motion estimation structure provided in this application; Figure 3 A schematic diagram of the structure of a motion estimation device provided in this application is shown. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0014] In some of the processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0015] For ease of reference, some terms used in this description are defined as follows. The terms presented and their respective definitions are not strictly limited to these definitions—a term may be further defined by its use in this disclosure. The term “example” as used herein means used as an example, instance, or illustration. Any aspect or design described herein as “exemplary” should not necessarily be construed as superior to other aspects or designs. Rather, the term “exemplary” is used to present the concept in a concrete manner. In this application and the appended claims, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise stated or clearly apparent from the context, “X uses A or B” is intended to mean any natural inclusive arrangement. That is, “X uses A or B” is satisfied if X employs A, X employs B, or X employs both A and B. As used herein, at least one of A or B means at least one of A, or at least one of B, or at least one of A and B. In other words, this phrase is disjunctive. The articles “a” and “an” used in this application and in the appended claims should generally be interpreted as “one or more”, unless otherwise stated or clearly apparent from the context that they are in the singular form.

[0016] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0017] Figure 1 A flowchart of a motion estimation method provided in this application is shown, as follows: Figure 1 As shown, the method may include the following steps: S101, Obtain the image block to be processed in the current image.

[0018] The current image can be any frame from the video to be processed. For example, when encoding a video stream, video images can be read frame by frame in chronological order, and the current frame to be encoded can be used as the current image for processing. Furthermore, the image data in the current image is typically divided into multiple image blocks of different sizes for prediction, transformation, and encoding processing. For example, the image blocks in the current image can include image blocks of different sizes such as 32×32, 16×16, and 8×8.

[0019] The image block to be processed can be an image block of any size divided from the current image.

[0020] In the embodiments of this application, motion estimation essentially involves finding the image region in the reference image that is most similar to the image patch to be processed in the current image. For example, in a video, a moving object typically changes position over time. Suppose the image patch to be processed in the current frame corresponds to a car moving to the right; in the previous frame of the reference image, the car might have been positioned further to the left, and in the next frame, it might have been positioned further to the right. The purpose of motion estimation is to find the car's original position in the reference image and determine its motion from that position to the current position, or to find the car's future position in the reference image and determine its motion from the current position to the future position. After determining the motion, the corresponding motion vectors can be used to encode the video. For example, only the number of pixels the image patch has moved relative to the reference image needs to be recorded, without needing to repeatedly encode the entire image content, thereby reducing the amount of video data and improving video compression efficiency.

[0021] S102, the first algorithm is used to estimate the motion of the image block to be processed, and the first motion vector corresponding to the image block to be processed is obtained.

[0022] The first algorithm can be a coarse motion estimation (CME) algorithm. Coarse motion estimation algorithms are typically used to quickly determine the approximate direction or location of motion of an image patch within a large search range. They can search the reference image at predetermined steps to reduce the number of searches and matching calculations. Compared to higher-precision motion estimation methods, coarse motion estimation algorithms have lower search accuracy but faster search speed, making them suitable as the initial motion search stage.

[0023] The first motion vector represents the initial motion result of the image patch to be processed in the reference image. Specifically, the first motion vector can include horizontal displacement information and vertical displacement information. For example, if the first motion vector is (4, 2), it means that relative to the corresponding region in the reference image, the image patch to be processed has moved 4 pixels in the positive horizontal direction and 2 pixels in the positive vertical direction. In other words, the first motion vector actually represents the approximate position of the image patch to be processed in the reference image, or the initial search position, providing a basis for subsequent higher-precision motion estimation.

[0024] In the embodiments of this application, during video encoding, the encoder typically divides the current image into blocks to form multiple image blocks of different sizes. For example, the current image can be divided into image blocks of different sizes, such as 32×32, 16×16, and 8×8. Subsequently, coarse motion estimation processing can be performed on the image blocks of different sizes to obtain the first motion vector corresponding to each image block. For example, for a 32×32 image block, the reference region most similar to the 32×32 image block can be searched in the reference image, and the corresponding first motion vector can be determined; for 16×16 and 8×8 image blocks, motion search can be performed in a similar manner to obtain the corresponding first motion vectors. It should be noted that the image block to be processed in this embodiment can be any one of the above multiple image blocks. An 8×8 image block can be used as the image block to be processed, and in the subsequent motion estimation process, it can be further combined with the motion information corresponding to other related image blocks (such as a 16×16 or 32×32 image block including the 8×8 image block) to perform subsequent high-precision motion search.

[0025] S103, based on the first motion vector and the second motion vector, determine a plurality of candidate search centers, wherein the second motion vector is the motion vector corresponding to at least one associated image block that has an association relationship with the image block to be processed.

[0026] The associated image block can represent an image block that has a predetermined association relationship with the image block to be processed. In some embodiments, the association relationship can include a spatial association relationship, such as a hierarchical containment relationship; it can also include a temporal or spatial adjacency relationship, such as a collocated block or a neighboring block. For example, the associated image block can represent an image block that has a hierarchical containment relationship with the image block to be processed, specifically it can be a larger image block that includes the image block to be processed. For example, when the image block to be processed is an 8×8 image block, the associated image block can be a 16×16 image block or a 32×32 image block that includes the 8×8 image block.

[0027] The second motion vector can be the motion vector corresponding to the associated image block. For example, when the image block to be processed is an 8×8 image block, the associated image block can be a 16×16 image block or a 32×32 image block that includes the 8×8 image block. After coarse motion estimation, the corresponding second motion vector can be obtained for the 16×16 image block or the 32×32 image block, respectively.

[0028] In this context, the candidate search center can be understood as the starting position used in subsequent high-precision motion search processes. Furthermore, the candidate search center can correspond to the pointing position of the motion vector in the reference image. In other words, the motion vector describes the displacement relationship from the current image patch position to the corresponding position in the reference image, and the candidate search center can correspond to the reference image position pointed to by this displacement relationship. For example, if the position corresponding to the first motion vector is A, and the positions corresponding to the second motion vectors are B and C, then A, B, and C can be used as candidate search centers for subsequent motion searches, respectively.

[0029] In the embodiments of this application, multiple different candidate search centers can be obtained by simultaneously utilizing the first motion vector corresponding to the image patch to be processed and the second motion vector corresponding to the associated image patch. Since image patches of different sizes usually have a certain motion correlation, the motion position corresponding to the associated image patch can provide additional motion search reference for the current image patch to be processed, thereby improving the efficiency of subsequent motion search.

[0030] S104. Based on multiple candidate search centers, the second algorithm is used to estimate the motion of the image block to be processed, and the third motion vector corresponding to the image block to be processed is obtained. The estimation accuracy of the second algorithm is higher than that of the first algorithm.

[0031] The second algorithm can be a refined motion estimation (RME) algorithm. Refined motion estimation algorithms are typically used to further refine the search for motion locations within a smaller search range. For example, the first motion vector obtained by a coarse motion estimation algorithm can only represent the approximate motion location of the image patch to be processed, while a refined motion estimation algorithm can perform a denser search around the candidate search center, thereby determining a more accurate motion location. Those skilled in the art will understand that higher estimation accuracy can mean that the second algorithm has a smaller search granularity, a more focused search range, more search locations, or more precise matching calculations. For example, the first algorithm can search with a larger step size, while the second algorithm can search with a smaller step size; or, the first algorithm is only used to obtain integer pixel-level motion locations, while the second algorithm can further determine more precise motion locations.

[0032] The third motion vector represents the motion estimation result after further optimization. Compared to the first motion vector, the third motion vector can usually more accurately reflect the true motion position of the image patch being processed in the reference image. For example, in a video scene where a person is moving, coarse motion estimation may only determine that the person is roughly moving to the right, while refined motion estimation can further accurately determine the actual position of the person's movement, thus obtaining a more accurate third motion vector.

[0033] In the embodiments of this application, after searching using a refined motion estimation algorithm based on multiple candidate search centers to obtain a third motion vector, subsequent more refined motion estimation and video encoding processes can be performed based on the third motion vector, thereby improving video encoding efficiency and encoding quality.

[0034] Through the above steps, the computer acquires the image block to be processed in the current image; uses a first algorithm to estimate the motion of the image block to be processed, and obtains the first motion vector corresponding to the image block to be processed; determines multiple candidate search centers based on the first motion vector and the second motion vector, wherein the second motion vector is the motion vector corresponding to at least one associated image block that has a relationship with the image block to be processed; and uses a second algorithm to estimate the motion of the image block to be processed based on the multiple candidate search centers, and obtains the third motion vector corresponding to the image block to be processed, wherein the estimation accuracy of the second algorithm is higher than that of the first algorithm. That is, a motion estimation algorithm with lower precision is first used to estimate the motion of the image block to be processed, and the first motion vector corresponding to the image block to be processed is obtained. Based on the first motion vector, the motion vectors corresponding to at least one associated image block associated with the image block to be processed can be combined, and a higher precision motion estimation algorithm can be used to estimate the motion of the image block to be processed. Since the associated image block is related to the image block to be processed, the precision of the motion estimation method can be improved. Moreover, the associated image block can be the image block whose corresponding motion vector has been calculated in the lower precision motion estimation algorithm. Therefore, the motion vector corresponding to the associated image block can be obtained more conveniently. This solves the technical problem of low estimation efficiency of motion estimation methods in related technologies and achieves the beneficial effect of improving estimation efficiency.

[0035] As an optional embodiment, based on multiple candidate search centers, a second algorithm is used to perform motion estimation on the image block to be processed to obtain a third motion vector corresponding to the image block to be processed, including: acquiring a reference image; determining the target candidate image block corresponding to each of the multiple candidate search centers in the reference image; selecting the target image block from the target candidate image blocks corresponding to each of the multiple candidate search centers; and determining the third motion vector based on the image block to be processed and the target image block.

[0036] The reference image can represent historical images used to perform motion estimation. For example, in video encoding, one or more frames preceding the current image can be used as the reference image. Since adjacent frames in a video typically have strong temporal continuity, most of the content in the current image can usually be found in the reference image.

[0037] The candidate search center can represent the starting position of the search used in the subsequent motion search process, which usually corresponds to the pointing position of the motion vector in the reference image.

[0038] Here, the target candidate image block can be represented as an image block that has a high degree of matching with the image block to be processed within the search range corresponding to a certain candidate search center.

[0039] The target image block can be represented as the optimal matching image block determined from multiple target candidate image blocks.

[0040] The third motion vector can represent the motion relationship between the image block to be processed and the target image block.

[0041] In this embodiment, multiple candidate search centers can be used as starting points to perform high-precision motion search in the reference image, thereby obtaining multiple search results. Then, the target image block that best matches the image block to be processed is further selected from the multiple search results, and a third motion vector is determined based on the target image block. For example, the first motion vector corresponding to the image block to be processed itself can point to position A in the reference image, while the second motion vector corresponding to the associated image block can point to positions B and C in the reference image, respectively. Positions A, B, and C can all serve as candidate search centers. When searching with position A as the center, the image block A1 most similar to the image block to be processed can be obtained; when searching with position B as the center, the most similar image block B1 can be obtained; when searching with position C as the center, the most similar image block C1 can be obtained. Therefore, A1, B1, and C1 can be used as target candidate image blocks corresponding to the multiple candidate search centers, and the target image block can be selected from A1, B1, and C1.

[0042] Since different candidate search centers may correspond to different motion directions or motion locations, using multiple candidate search centers simultaneously can increase the range of motion areas covered by the search process, thereby improving the accuracy of motion estimation and the efficiency of motion search.

[0043] As an optional embodiment, determining the target candidate image block corresponding to each of the multiple candidate search centers in the reference image includes: determining the search region corresponding to each of the multiple candidate search centers in the reference image; for any candidate search center included in the multiple candidate search centers, determining the target candidate image block corresponding to the candidate search center in the following manner: determining multiple candidate reference image blocks in the search region corresponding to the candidate search center; determining the matching cost between the image block to be processed and the multiple candidate reference image blocks respectively; and determining the target candidate image block corresponding to the candidate search center among the multiple candidate reference image blocks based on the matching cost between the image block to be processed and the multiple candidate reference image blocks.

[0044] The search region can represent a local search area defined around the candidate search center. For example, the search region can be defined within the pixel range surrounding the candidate search center.

[0045] Among them, candidate reference image blocks can represent multiple image blocks to be matched in the search area.

[0046] The matching cost value represents the degree of similarity between the image patch to be processed and the candidate reference image patch. For example, the matching cost value can be calculated using methods such as the sum of absolute errors (SAD) and the sum of squared errors (SSD). Generally, the smaller the matching cost value, the more similar the two image patches are.

[0047] In this embodiment, the location corresponding to the candidate search center is not directly used as the final search result. Instead, a local fine-grained search is performed around each candidate search center. For example, near a certain candidate search center, there may be multiple candidate reference image blocks at different locations, and the degree of matching between these candidate reference image blocks and the image block to be processed may not be the same. Therefore, the matching cost between the image block to be processed and each candidate reference image block can be calculated separately, and the image block with the better matching effect can be selected as the target candidate image block corresponding to the candidate search center. For example, candidate search centers may include positions A, B, and C. When searching with position A as the center, candidate reference image blocks A1, A2, and A3 can be obtained. The matching cost between the image block to be processed and A1, A2, and A3 is calculated respectively, and A1 is selected as the most similar target candidate image block. When searching with position B as the center, candidate reference image blocks B1, B2, and B3 can be obtained. The matching cost between the image block to be processed and B1, B2, and B3 is calculated respectively, and B1 is selected as the most similar target candidate image block. When searching with position C as the center, candidate reference image blocks C1, C2, and C3 can be obtained. The matching cost between the image block to be processed and C1, C2, and C3 is calculated respectively, and C1 is selected as the most similar target candidate image block.

[0048] By using the above method, each candidate search center can obtain the corresponding locally optimal search result, thereby improving the accuracy of the subsequent global optimization process.

[0049] As an optional embodiment, the target candidate image block corresponding to the candidate search center is determined among the multiple candidate reference image blocks based on the matching cost value between the image block to be processed and multiple candidate reference image blocks, including: determining the candidate reference image block with the lowest matching cost value among the multiple candidate reference image blocks as the target candidate image block corresponding to the candidate search center.

[0050] In this embodiment, the image block with the lowest matching value, i.e., the one with the highest similarity to the image block to be processed, can be selected from multiple candidate reference image blocks. For example, in the search area corresponding to a certain candidate search center, there are candidate reference image blocks A1, A2, and A3, and the matching values ​​of the image block to be processed and their corresponding reference image blocks are 80, 150, and 120, respectively. Since the matching value of candidate reference image block A1 is the lowest, candidate reference image block A1 can be determined as the target candidate image block corresponding to the candidate search center.

[0051] In this embodiment, by using the lowest matching cost as the criterion for determining the target candidate image block, the similarity between image blocks can be evaluated using a quantitative method, thereby improving the accuracy and stability of the target candidate image block selection process.

[0052] As an optional embodiment, selecting a target image block from the target candidate image blocks corresponding to each of the multiple candidate search centers includes: determining the matching cost between the target candidate image blocks corresponding to each of the multiple candidate search centers and the image block to be processed; and determining the target candidate image block with the lowest matching cost among the target candidate image blocks corresponding to each of the multiple candidate search centers as the target image block.

[0053] In this context, the target candidate image patches corresponding to each of the multiple candidate search centers can be understood as multiple locally optimal results obtained after performing local searches on each candidate search center. The target image patch can then represent the globally optimal result among these locally optimal results. For example, after performing local searches on candidate search centers A, B, and C respectively, target candidate image patches A1, B1, and C1 can be obtained. Subsequently, the matching cost between the image patch to be processed and A1, B1, and C1 can be calculated separately, and the target candidate image patch with the lowest matching cost can be selected as the final target image patch.

[0054] In other words, the embodiments of this application actually include a two-stage screening process: in the first stage, locally optimal results are determined within the local search regions corresponding to each candidate search center; in the second stage, globally optimal results are further determined among multiple locally optimal results. This approach avoids relying solely on a single search center for motion estimation, thereby improving the accuracy and robustness of the final motion search results.

[0055] As an optional embodiment, the second motion vector is determined by: acquiring at least one parent image block corresponding to the same region as the image block to be processed, and the motion vector corresponding to the at least one parent image block; determining that the at least one parent image block is at least one associated image block, and determining that the motion vector corresponding to the at least one parent image block is the second motion vector.

[0056] In this context, the parent image block can be a larger image block within the image block partitioning hierarchy that contains the image block to be processed. In other words, the image block to be processed can be understood as a sub-region within the parent image block. For example, when the image block to be processed is an 8×8 image block, the parent image block corresponding to the same region as the 8×8 image block can be a 16×16 image block or a 32×32 image block that contains the 8×8 image block. In this case, the 16×16 image block or the 32×32 image block can serve as the parent image block corresponding to the image block to be processed.

[0057] In this context, "corresponding to the same region" means that the parent image block and the image block to be processed have an inclusion relationship or a region overlap relationship in the image space. For example, the image block to be processed is located inside the image region covered by the parent image block.

[0058] The motion vector corresponding to the parent image block represents the motion result obtained after motion estimation of the parent image block. In some embodiments, the motion vector corresponding to the parent image block can be the motion vector determined during the previous coarse motion estimation process. For example, after performing coarse motion estimation on multiple image blocks of different sizes using the first algorithm, motion vectors corresponding to 32×32 image blocks, 16×16 image blocks, and 8×8 image blocks can be obtained respectively. The motion vector corresponding to the larger image block can be used as the motion vector corresponding to the parent image block in this embodiment.

[0059] Here, the associated image block can represent an image block that has a predetermined association relationship with the image block to be processed. In some embodiments, the association relationship can include a spatial association relationship; for example, the associated image block can be a larger image block that includes the image block to be processed. In this embodiment, the associated image block can be a parent image block. Correspondingly, the motion vector corresponding to the parent image block can be determined as a second motion vector.

[0060] Because adjacent regions or image blocks of different sizes within the same region often exhibit a certain correlation in their direction and trend of motion. For example, when a person in a video moves to the right, the 32×32, 16×16, and 8×8 image blocks containing the person's area will typically show similar directions of motion. Therefore, compared to using only the motion vector corresponding to the image block to be processed itself, this embodiment further introduces the motion vector corresponding to the parent image block as a second motion vector, thereby providing additional search reference for subsequent high-precision motion search. Since the parent image block usually covers a larger image area, its motion estimation results often have better overall stability. Utilizing this motion information can improve the effectiveness of candidate search centers, thereby reducing the invalid search range, improving motion estimation efficiency, and increasing the accuracy of the final motion estimation result. In other words, this embodiment actually uses the motion vector corresponding to the larger area where the image block to be processed is located, which has already been determined in the coarse motion estimation process, to assist the image block to be processed in performing subsequent high-precision motion search.

[0061] As an optional embodiment, after the motion estimation method uses a second algorithm to estimate the motion of the image block to be processed based on multiple candidate search centers to obtain a third motion vector corresponding to the image block to be processed, it further includes: using a third algorithm to estimate the motion of the image block to be processed based on the third motion vector to obtain a fourth motion vector corresponding to the image block to be processed. The estimation accuracy of the third algorithm is higher than that of the second algorithm. When using the third algorithm to estimate the motion of the image block to be processed, multiple candidate search centers can also be determined based on the third motion vector of the image block to be processed and the motion vectors of its associated image blocks.

[0062] The third algorithm can be a fractional motion estimation (FME) algorithm. FME algorithms are typically used to determine more precise motion positions based on already obtained high-precision positions. Compared to previous motion estimation stages, the third algorithm offers higher search accuracy and finer search granularity.

[0063] The third motion vector can represent the motion estimation result determined by the second algorithm. The second algorithm can be a refined motion estimation (RME) algorithm, which is typically able to determine the motion position at the integer pixel level. Correspondingly, the third motion vector can represent the integer pixel-level motion result of the image patch to be processed in the reference image.

[0064] The fourth motion vector represents the further optimized motion estimation result. Compared to the third motion vector, the fourth motion vector can usually more accurately characterize the actual motion position of the image patch in the reference image. In some scenarios, the true motion position of the image patch may not correspond exactly to an integer pixel position, but may be located between two pixels. In this case, using only integer pixel-level motion estimation may not accurately describe the true motion. Therefore, a fractional motion estimation algorithm can be further used to perform a more refined search near the integer pixel position. For example, the third motion vector may be (5, 2), indicating that the image patch moves 5 pixels horizontally and 2 pixels vertically. After fractional motion estimation, the fourth motion vector may be further determined to be a more refined motion position such as (5.25, 2.5) or (5.5, 2.25).

[0065] The higher estimation accuracy of the third algorithm compared to the second algorithm indicates that the third algorithm has a smaller search step size, finer search granularity, more search locations, or higher accuracy in describing motion positions. For example, the second algorithm may only support integer pixel-level search, while the third algorithm may support half-pixel-level, quarter-pixel-level, or even higher precision search.

[0066] In this embodiment, after obtaining the third motion vector, motion estimation does not end directly, but rather a higher-precision motion search is performed based on the third motion vector. Furthermore, the motion search logic with multiple candidate search centers described above can be reused during the motion estimation process using the third algorithm. For example, for a smaller image patch to be processed, in addition to using the third motion vector corresponding to the image patch itself as the search center, the motion vector corresponding to the parent image patch or a larger region image patch can be used as an auxiliary search center. For example, when the image patch to be processed is an 8×8 image patch, the third motion vector corresponding to the 8×8 image patch, as well as the motion vector corresponding to a 16×16 or 32×32 image patch, can be used simultaneously to determine multiple candidate search centers, and fractional motion search can be performed around each of these candidate search centers.

[0067] As an optional embodiment, during motion estimation by the hardware encoder, the processes of "reference pixel acquisition," "matching cost calculation," and "result comparison" can be separated and executed by independent hardware modules. That is, by using independent hardware modules to perform reference pixel acquisition, matching cost calculation, and result comparison, the acquired reference pixel data can be reused during the search process for different candidate search centers. Alternatively, a first hardware module can acquire a reference image, a second hardware module can determine the matching cost between the image block to be processed and multiple candidate reference image blocks, and a third hardware module can determine the candidate reference image block with the lowest matching cost among the multiple candidate reference image blocks as the target candidate image block corresponding to the candidate search center. The first, second, and third hardware modules are independent of each other.

[0068] The process involves several steps: First, obtaining reference pixels involves reading pixel data corresponding to candidate reference image blocks from a reference image. Second, cost calculation involves calculating the matching cost based on the pixel differences between the image block to be processed and the candidate reference image blocks. Third, result comparison involves comparing the matching costs corresponding to multiple candidate search results and determining the final motion search result.

[0069] For example, when the image block to be processed is an 8×8 image block, motion search can be performed based on multiple candidate search centers. Since there may be some identical or overlapping reference pixel regions during the search process corresponding to multiple candidate search centers, this embodiment can reuse the already acquired reference pixel data without having to repeatedly perform reference pixel reading operations for different candidate search centers. That is, after completing one reference pixel acquisition, the acquired reference pixel data can be provided to multiple cost calculation modules simultaneously, thereby reducing repeated memory access operations and the number of reference pixel reads.

[0070] Furthermore, for multiple candidate search centers corresponding to the same block size, the corresponding cost calculation process can be executed in parallel. For example, for multiple 8×8 image block search tasks, the matching cost between the image block to be processed and multiple candidate reference image blocks can be calculated simultaneously without sequential calculation.

[0071] In this embodiment, by decoupling the reference pixel acquisition process from the cost calculation process and combining reference pixel reuse and parallel cost calculation, the memory access pressure in the hardware encoder can be reduced, the hardware resource consumption caused by repeated data reading can be reduced, and the processing efficiency of the motion estimation process can be improved, thereby reducing hardware overhead and improving video coding performance.

[0072] As one specific embodiment, such as Figure 2 As shown, the motion estimation method in this embodiment may include a coarse motion estimation (CME) stage, a refined motion estimation (RME) stage, and a fractional motion estimation (FME) stage. In particular, by introducing motion vectors corresponding to image blocks of different sizes as candidate search centers during the high-precision motion estimation process, the efficiency of motion search and the accuracy of motion estimation can be improved.

[0073] Specifically, at the beginning of motion estimation, coarse motion estimation (CME) can be performed based on the current image. Coarse motion search can be performed separately for 32×32, 16×16, and 8×8 image blocks, yielding coarse motion estimation results for the corresponding image block sizes: the optimal CME output mv for 32×32, 16×16, and 8×8 image blocks, respectively. Since coarse motion estimation is primarily used to quickly determine the approximate motion position of image blocks, this stage typically employs a lower-precision search method to reduce the number of searches and matching computations.

[0074] After obtaining the coarse motion vectors corresponding to image patches of different sizes, the fine motion estimation (RME) stage can be entered. Unlike the traditional approach that only uses the motion vector of the current patch itself as the search center, this embodiment utilizes motion vectors corresponding to multiple image patches of different sizes simultaneously as candidate search centers for subsequent motion searches. For example, for the motion estimation process of an 8×8 image patch, not only can the CME motion vector corresponding to the 8×8 image patch itself be used as the search center, but the CME motion vectors corresponding to 16×16 or 32×32 image patches can also be used as auxiliary search centers. In other words, in this embodiment, motion information corresponding to larger image patches can also participate in the subsequent high-precision motion search of smaller image patches.

[0075] Furthermore, after identifying multiple candidate search centers, corresponding reference pixel buffers can be obtained for each candidate search center. For example, reference pixel data for the corresponding search region can be read from the reference image around different search centers, and subsequent motion estimation calculations can be performed based on the obtained reference pixel data. Since some reference pixel regions corresponding to multiple search centers may overlap, in this embodiment, the reference pixel data can be reused based on the hardware architecture without repeatedly performing the reference pixel reading operation, thereby reducing additional hardware overhead.

[0076] Subsequently, refined motion estimation can be performed separately for image blocks of different sizes. For example, for a 32×32 coding unit, corresponding motion estimation can be performed based on a 32×32 search center; for a 16×16 coding unit, multiple optimal 16×16 search results can be obtained simultaneously based on different search centers; and for an 8×8 coding unit, multiple optimal 8×8 search results can be obtained separately based on multiple different search centers.

[0077] After completing the RME stage, the optimal 32×32 output mv, optimal 16×16 output mv, and optimal 8×8 output mv can be output respectively. These motion vectors can be used as inputs for the next stage of higher-precision motion estimation.

[0078] Furthermore, the subsequent FME (Fractional Motion Estimation) stage can employ similar processing logic to the RME stage. Specifically, it can continue to perform higher-precision motion searches based on multiple candidate search centers. For example, based on the integer pixel-level motion estimation results, it can further perform half-pixel or quarter-pixel-level motion searches to obtain more refined motion estimation results. Simultaneously, in the FME stage, the motion vectors corresponding to larger image regions can also be used to assist the search of smaller image regions, further improving the accuracy of the motion search.

[0079] In this embodiment, by introducing motion vectors corresponding to multiple image blocks of different sizes as candidate search centers during high-precision motion estimation, the coverage of motion search can be improved, and the probability of the search getting trapped in local optima can be reduced. Simultaneously, by reusing the reference pixel buffer and splitting and parallelizing the processes of reference pixel acquisition, cost calculation, and result comparison, redundant memory access operations and hardware resource consumption can be reduced. This improves motion estimation accuracy while reducing hardware overhead and increasing video coding efficiency. Furthermore, the scheme provided in this application can be applied to the preprocessing Pass stage of multi-pass video coding. Experimental results show that the optimized motion estimation results obtained in this stage can be effectively transferred to subsequent actual coding Pass stages, continuously improving overall coding efficiency and demonstrating significant practical value.

[0080] According to an embodiment of this application, a motion estimation device is provided. Figure 3 A schematic diagram of the structure of a motion estimation device provided in this application is shown, as follows: Figure 3 As shown, the device includes: an acquisition module 301, a first estimation module 302, a determination module 303, and a second estimation module 304. The following is a description of the device.

[0081] The acquisition module 301 is used to acquire the image block to be processed in the current image; The first estimation module 302 is connected to the acquisition module 301 and is used to perform motion estimation on the image block to be processed using a first algorithm to obtain the first motion vector corresponding to the image block to be processed. The determination module 303, connected to the first estimation module 302, is used to determine multiple candidate search centers based on the first motion vector and the second motion vector, wherein the second motion vector is the motion vector corresponding to at least one associated image block that has an association relationship with the image block to be processed; The second estimation module 304, connected to the determination module 303, is used to perform motion estimation on the image block to be processed based on multiple candidate search centers using a second algorithm, to obtain the third motion vector corresponding to the image block to be processed. The estimation accuracy of the second algorithm is higher than that of the first algorithm. Among them, the hardware components corresponding to the acquisition module, the hardware components corresponding to the first estimation module, the hardware components corresponding to the determination module, and the hardware components corresponding to the second estimation module are independent of each other, so as to process motion estimation of the image block to be processed based on different candidate search centers in parallel. The reference pixel data acquired by the acquisition module is reused for the matching cost calculation of multiple candidate search centers.

[0082] It should be noted that the acquisition module 301, the first estimation module 302, the determination module 303, and the second estimation module 304 mentioned above correspond to steps S101 to S104 in the embodiments. Multiple modules and their corresponding steps implement the same instances and application scenarios, and their implementation principles and technical effects will not be repeated here. The specific methods by which each module and unit in the device of the above embodiments performs operations have been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0083] Optionally, each of the above modules can be implemented by independent hardware circuits. The reference pixel acquisition unit, the cost value calculation unit, and the result comparison unit are independent of each other to support parallel processing for different candidate search centers. In addition, the reference pixel data acquired by the reference pixel acquisition unit can be reused. That is, when the cost value calculation unit calculates the cost value based on multiple candidate search centers, the reference pixel data acquired by the reference pixel acquisition unit can be reused, reducing repeated memory access operations and hardware resource consumption.

[0084] This application embodiment also provides a computing device, which may include a storage component and a processing component; The storage component stores one or more computer instructions, wherein the one or more computer instructions are invoked and executed by the processing component to implement the method described in any of the above-mentioned embodiments.

[0085] Of course, computing devices may also include other components, such as input / output interfaces, display components, communication components, etc.

[0086] Input / output interfaces provide interfaces between processing components and peripheral interface modules, which can be output devices, input devices, etc. Communication components are configured to facilitate wired or wireless communication between computing devices and other devices.

[0087] The processing component may include one or more processors to execute computer instructions to complete all or part of the steps in the above-described method. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0088] Storage components are configured to store various types of data to support operations on the terminal. Storage components can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0089] The display component can be an electroluminescent (EL) element, a liquid crystal display or a microdisplay with a similar structure, or a retina-direct display or a similar laser scanning display.

[0090] It should be noted that the aforementioned computing device implementation method or processing method can be a physical device or an elastic computing host provided by a cloud computing platform. It can be implemented as a distributed cluster composed of multiple servers or terminal devices, or as a single server or a single terminal device.

[0091] When the above-mentioned computing device implements the above method, it can be specifically implemented as an electronic device. An electronic device can refer to a device used by a user that has the computing, Internet access, communication and other functions required by the user, such as a mobile phone, tablet computer, personal computer, wearable device, etc.

[0092] It should be noted that the aforementioned computing devices can be physical devices or elastic computing hosts provided by cloud computing platforms. They can be implemented as a distributed cluster of multiple servers or terminal devices, or as a single server or a single terminal device.

[0093] This application also provides a computer-readable storage medium storing a computer program that, when executed by a computer, can implement the above-described method. This computer-readable medium may be included in the electronic device described in the above embodiments; alternatively, it may exist independently and not be assembled into the electronic device.

[0094] This application also provides a computer program product comprising a computer program carried on a computer-readable storage medium, which, when executed by a computer, can implement the methods described above. In such an embodiment, the computer program may be downloaded and installed from a network, and / or installed from a removable medium. When the computer program is executed by a processor, it performs the various functions defined in the system of this application.

[0095] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0097] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computing device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A motion estimation method, characterized in that, include: Get the image patch to be processed in the current image; The first algorithm is used to perform motion estimation on the image block to be processed, and the first motion vector corresponding to the image block to be processed is obtained. Based on the first motion vector and the second motion vector, multiple candidate search centers are determined, wherein the second motion vector is the motion vector corresponding to at least one associated image block that has an association with the image block to be processed; Based on the multiple candidate search centers, a second algorithm is used to perform motion estimation on the image block to be processed, thereby obtaining a third motion vector corresponding to the image block to be processed. The estimation accuracy of the second algorithm is higher than that of the first algorithm.

2. The method according to claim 1, characterized in that, Based on the multiple candidate search centers, the second algorithm is used to perform motion estimation on the image block to be processed, obtaining the third motion vector corresponding to the image block to be processed, including: Obtain a reference image; In the reference image, the target candidate image patch corresponding to each of the plurality of candidate search centers is determined; Select a target image block from the target candidate image blocks corresponding to each of the multiple candidate search centers; The third motion vector is determined based on the image block to be processed and the target image block.

3. The method according to claim 2, characterized in that, The step of determining the target candidate image patch corresponding to each of the plurality of candidate search centers in the reference image includes: In the reference image, the search area corresponding to each of the plurality of candidate search centers is determined; For any candidate search center included in the plurality of candidate search centers, the target candidate image block corresponding to the candidate search center is determined in the following manner: multiple candidate reference image blocks are determined in the search area corresponding to the candidate search center; the matching value between the image block to be processed and the multiple candidate reference image blocks is determined respectively; based on the matching value between the image block to be processed and the multiple candidate reference image blocks, the target candidate image block corresponding to the candidate search center is determined among the multiple candidate reference image blocks.

4. The method according to claim 3, characterized in that, The step of determining the target candidate image block corresponding to the candidate search center among the plurality of candidate reference image blocks based on the matching value between the image block to be processed and the plurality of candidate reference image blocks includes: Among the multiple candidate reference image blocks, the candidate reference image block with the lowest matching cost is determined as the target candidate image block corresponding to the candidate search center.

5. The method according to claim 2, characterized in that, The step of selecting a target image block from the target candidate image blocks corresponding to each of the plurality of candidate search centers includes: The matching value between the target candidate image block and the image block to be processed corresponding to each of the plurality of candidate search centers is determined respectively; Among the target candidate image blocks corresponding to each of the multiple candidate search centers, the target candidate image block with the lowest matching cost is determined as the target image block.

6. The method according to claim 1, characterized in that, The second motion vector is determined in the following way: Obtain at least one parent image block that corresponds to the same region as the image block to be processed, and the motion vector corresponding to the at least one parent image block; The at least one parent image block is determined to be the at least one associated image block, and the motion vector corresponding to the at least one parent image block is determined to be the second motion vector.

7. The method according to any one of claims 1 to 6, characterized in that, Also includes: Based on the third motion vector, a third algorithm is used to estimate the motion of the image block to be processed, thereby obtaining a fourth motion vector corresponding to the image block to be processed. The estimation accuracy of the third algorithm is higher than that of the second algorithm.

8. A motion estimation device, characterized in that, include: The acquisition module is used to acquire the image patch to be processed in the current image; The first estimation module is used to perform motion estimation on the image block to be processed using a first algorithm to obtain a first motion vector corresponding to the image block to be processed. The determining module is used to determine multiple candidate search centers based on the first motion vector and the second motion vector, wherein the second motion vector is the motion vector corresponding to at least one associated image block that has an association relationship with the image block to be processed; The second estimation module is used to perform motion estimation on the image block to be processed based on the multiple candidate search centers using a second algorithm to obtain a third motion vector corresponding to the image block to be processed, wherein the estimation accuracy of the second algorithm is higher than that of the first algorithm. The hardware components corresponding to the acquisition module, the first estimation module, the determination module, and the second estimation module are independent of each other, so as to process motion estimation of the image block to be processed based on different candidate search centers in parallel. The reference pixel data acquired by the acquisition module is reused for the matching cost calculation of the multiple candidate search centers.

9. A computing device, characterized in that, This includes processing components and storage components; The storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement the motion estimation method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processing component, implements the motion estimation method as described in any one of claims 1 to 7.

Images