Image processing method for motion estimation

Abstract

An image processing method is proposed. The method includes to receive an I frame and generate a coordinate I-list corresponding to a first detected object in the I-frame and including a first coordinate candidate set having four coordinates, receive a P-frame following the I-frame and generate a coordinate P-list corresponding to a second detected object in the P-frame and including a second coordinate candidate set having four coordinates, generate at least one object region and an initial motion vector thereof of an interpolated frame between the I-frame and the P-frame based on the coordinate I-list and the coordinate P-list, and correct the initial motion vector to generate a corrected motion vector of the at least one object region of the interpolated frame.

Description

Technical Field

[0001] This invention relates to image processing techniques for motion estimation. Prior Technology

[0002] Typical non-AI-based motion vector estimation algorithms rely on the sum of absolute differences (SAD) between details and corresponding image blocks for object detection and motion vector convergence. However, this approach requires two P-frames for interpolation and incurs significant hardware costs for data storage. On the other hand, AI-based motion vector estimation algorithms for object detection converge based on object coordinate extraction. However, this method cannot guarantee information from a second P-frame, leading to interpolation failure. Furthermore, object coordinate extraction is less accurate in scenes with fast-moving small objects, resulting in inaccurate motion vector estimation. Summary of the Invention

[0003] To address the aforementioned problems, this invention proposes an image processing method for motion estimation.

[0004] In one embodiment of the present invention, the method includes receiving an I-frame and generating a coordinate I list that corresponds to a first detected object in the I-frame and has a first coordinate candidate set having four coordinates; receiving a P-frame following the I-frame and generating a coordinate P list that corresponds to a second detected object in the P-frame and includes a second coordinate candidate set having four coordinates; generating at least one object region and its initial motion vector of an interpolated frame between the I-frame and the P-frame based on the coordinate I list and the coordinate P list; and correcting the initial motion vector to generate a corrected motion vector of at least one object region of the interpolated frame. Simple Explanation of the Diagram

[0005] Figure 1 is a schematic diagram of an image processing apparatus according to an embodiment of the present invention. Figure 2 is a flowchart illustrating an image processing method according to an embodiment of the present invention. Figure 3 is a schematic diagram illustrating a coordinate processing method according to an embodiment of the present invention. Figures 4A and 4B illustrate the method of adding additional motion vectors according to an embodiment of the present invention. Implementation

[0006] Some exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. The number of elements shown in the drawings is for illustrative purposes only, and the present invention is not intended to be limited thereto. Furthermore, these embodiments are only a part of the present invention and do not disclose all possible implementations of the present invention. More precisely, these embodiments are merely examples of the methods within the scope of the present invention's patent application.

[0007] Figure 1 is a schematic diagram of an image processing apparatus according to an embodiment of the present invention. First, Figure 1 introduces the various components and their configuration in the computing device; detailed functions will be disclosed together with Figure 2.

[0008] Referring to Figure 1, the image processing apparatus 100 of this embodiment includes a memory 110 and a processor 120. The image processing apparatus 100 can be an electronic system or a computer system. The memory 110 can be various forms of random access memory (RAM), such as dynamic random access memory (DRAM). The processor 120 can be one or more northbridges, southbridges, field-programmable arrays (FPGAs), programmable logic devices (PLDs), application-specific integrated circuits (ASICs), other similar devices, integrated circuits, or combinations thereof. The processor 120 can also be a central processing unit (CPU), a programmable general-purpose or special-purpose microprocessor, a digital signal processor (DSP), a graphics processing unit (GPU), other similar devices, integrated circuits, or combinations thereof.

[0009] Figure 2 is a flowchart illustrating an image processing method according to an embodiment of the present invention, wherein the steps in Figure 2 can be implemented by the image processing apparatus 100 in Figure 1.

[0010] In this embodiment, it is assumed that video frames with a time sequence, including fast-moving small objects (e.g., a ball), are continuously received by the image processing device 100. Furthermore, it is assumed that an AI-based model (e.g., a CNN model) capable of recognizing specific objects or patterns has been pre-trained and pre-stored in memory 110.

[0011] Referring to Figures 2 and 1, the processor 120 of the image processing apparatus 100 receives I-frames 210 from a time-series video frame and generates a coordinate I-list 220. Here, the coordinate I-list 220 includes at least one set of coordinate candidates, and each set of coordinate candidates includes four coordinates. Each set of coordinate candidates represents a region of interest, which includes detected objects identified from the I-frame 210 by an AI-based model, and the four coordinates represent the four vertices of the region of interest. In another embodiment, each set of coordinate candidates may include more than four coordinates to provide more information about the region of interest; this invention is not limited thereto.

[0012] Next, the processor 120 of the image processing apparatus 100 receives a P-frame 230 following the I-frame 210 from the time-series video frames and generates a coordinate P-list 240 in a similar manner. That is, the coordinate P-list 240 includes at least one set of coordinate candidates, and each set of coordinate candidates includes four coordinates. Each set of coordinate candidates represents a region of interest, which includes detected objects identified from the P-frame 230 by an AI-based model, and the four coordinates represent the four vertices of the region of interest.

[0013] Next, the processor 120 generates at least one object region and its initial motion vector 250 for the interpolated frame PH between I-frame 210 and P-frame 230 based on the coordinate I list 220 and the coordinate P list 240. It should be noted that since I-frame 210 and P-frame 230 are processed independently, the coordinate I list 220 and the coordinate P list 240 may not have a one-to-one correspondence. Therefore, the processor 120 performs coordinate processing on the coordinate I list 220 and the coordinate P list 240 to obtain at least one correspondence by performing matching on the coordinate I list 220 and the coordinate P list 240.

[0014] In this embodiment, the processor 120 uses a coordinate list with a small number of coordinate candidate sets as the primary list to perform matching. For example, FIG3 is a schematic diagram illustrating a coordinate processing method according to an embodiment of the present invention.

[0015] Referring to Figures 3 and 1, coordinate list 320 includes three candidate coordinate sets coor0, coor1, and coor2, while coordinate list 340 includes two candidate coordinate sets coor3 and coor4. Since the number of candidate coordinate sets in coordinate list 340 is less than the number of candidate coordinate sets in coordinate list 320, processor 120 treats coordinate list 340 as the primary list and searches for the candidate coordinate set with the minimum distance from coordinate list 320 to candidate coordinate sets coor3 and coor4 in coordinate list 340.

[0016] In this example, the candidate coordinate sets coor1 and coor2 in coordinate list 320 are the candidate coordinate sets with the minimum distance to the candidate coordinate sets coor3 and coor4 in coordinate list 340, respectively. Candidate coordinate sets coor1 and coor3 form one correspondence, while candidate coordinate sets coor2 and coor4 form another. Here, the unmatched candidate coordinate set coor0 can be considered a false detection and can be ignored.

[0017] In another example, when the number of candidate coordinate sets in list P is greater than the number of candidate coordinate sets in list I, processor 120 treats list I as the primary list and searches list P for the candidate coordinate set with the minimum distance to each candidate coordinate set in list I. Those skilled in the art should be able to deduce the details from Figure 3, and will not repeat them here.

[0018] Please refer back to Figure 2. For clarity, one of the correspondences (hereinafter referred to as the "specified correspondence") will be explained below. This specified correspondence includes a "first coordinate candidate set" from coordinate list 220 and a "second coordinate candidate set" from coordinate list 240. For the interpolated frame PH, the processor 120 calculates four motion vectors (hereinafter referred to as "coordinate motion vectors") corresponding to the four pairs of coordinates in the specified correspondence, and generates a specified object region for the interpolated frame PH based on the first coordinate candidate set and the four coordinate motion vectors. Here, the specified object region of the interpolated frame PH corresponds to the region of interest in both the I-frame and P-frame. The processor 120 also calculates the initial motion vectors for the specified object region of the interpolated frame PH based on the four coordinate motion vectors.

[0019] In detail, the four coordinate motion vectors that specify the correspondence can be expressed as the following equation (1): mv_x0 = I_left–P_left mv_x1 = I_right–P_right mv_y0 = I_top–P_top mv_y1 = I_bottom–P_bottom Here, mv_x0, mv_x1, mv_y0, and mv_y1 represent four coordinate motion vectors, I_left, I_right, I_top, and I_bottom represent the first specified coordinate set in coordinate list I 220, and P_left, P_right, P_top, and P_bottom represent the second specified coordinate set in coordinate list P 240.

[0020] Furthermore, the specified object region of the interpolated frame PH can be represented by the following equation (2): PH_Left = I_Left – int(ph_k × mv_x0 / 128.0) PH_Right = I_Right – int(ph_k × mv_x1 / 128.0) PH_Top = I_Top – int(ph_k × mv_y0 / 128.0) PH_Bottom = I_Bottom – int(ph_k × mv_y1 / 128.0) Here, `int` represents integer operations. `PH_Left`, `PH_Right`, `PH_Top`, and `PH_Bottom` represent the four vertices of the specified object region of the interpolated frame `PH`. It should be noted that `ph_k / 128.0` represents the phase of the interpolated frame `PH`. For example, if the phase of the interpolated frame `PH` is 1 / 4, then `ph_k` equals 32; if the phase of the interpolated frame `PH` is 2 / 4, then `ph_k` equals 64.

[0021] Next, the initial motion vector of the specified object region of the interpolated frame PH can be expressed as the following equation (3): MV_X = min / max / avg(mv_x0, mv_x1) MV_Y = min / max / avg(mv_y0, mv_y1) Here, MV_X represents the horizontal element of the initial motion vector, which can be the minimum, maximum, or average of two horizontal motion vectors, while MV_Y represents the vertical element of the initial motion vector, which can be the minimum, maximum, or average of two vertical motion vectors.

[0022] For accuracy and completeness, the processor 120 performs motion vector correction on the initial motion vector to generate a corrected motion vector 260 for the object region of the interpolated frame PH, thereby reducing the likelihood of the AI-based model detecting inaccurate results from I-frames or P-frames. In this embodiment, the processor 120 adds multiple additional motion vectors to the vicinity of the initial motion vector and selects the optimal motion vector that minimizes a predetermined cost function from the initial motion vector and the additional motion vectors as the corrected motion vector 260, where the predetermined cost function can be any cost function in the prior art.

[0023] For example, Figures 4A and 4B illustrate the addition of additional motion vectors according to an embodiment of the present invention.

[0024] Referring to Figure 4A, when the phase of the interpolated frame is 0 (i.e., I-frame), four additional motion vectors A1, A2, A3, and A4 will be added to the neighboring positions of the initial motion vector A = (x, y) on the interpolated frame, where: A1 = (x – th1, y – th1) A2 = (x + th1, y – th1) A3 = (x – th1, y + th1) A4 = (x + th1, y + th1) Here, the value of th1 is a preset constant. Furthermore, four additional motion vectors A5, A6, A7, and A8 can be added to the midpoints between the initial motion vector and the additional motion vectors A1, A2, A3, and A4, respectively. From another perspective, the additional motion vectors A1 to A8 and the initial motion vector A of the interpolated image frame form two concentric squares.

[0025] Referring to Figure 4B, when the phase of the interpolated frame is not 0, eight additional motion vectors B1~B8 will be added to the vicinity of the initial motion vector B = (x, y) of the interpolated frame, where: B1 = (x – th2, y – th2) B2 = (x + th2, y – th2) B3 = (x + th2, y + th2) B4 = (x – th2, y + th2) Here, the value of th2 is a different preset constant than th1 and can vary based on the phase of the interpolated frame. For example, the closer the interpolated frame is to the P-frame (i.e., the larger the phase), the smaller the spacing between the extra motion vectors (i.e., the smaller the value of th2). Furthermore, extra motion vectors B5, B6, B7, and B8 are added at the midpoint between every two extra motion vectors B1 to B4. From another perspective, the extra motion vectors B1 to B8 are equidistantly distributed and form a square around the initial motion vector MV of the interpolated frame.

[0026] The optimal motion vector (i.e., the corrected motion vector) from the initial motion vector and additional motion vectors enables smoother and more fluid motion in the video. Overall, the image processing method proposed in this invention solves the problem of poor motion vector convergence in existing technologies, and achieves effective motion estimation, particularly for videos with fast-moving objects.

[0027] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims and their equivalents.

[0028] 100: Image processing device 110: Memory 120: Processor 210:I frame 220, 320: Coordinate I List 230:P frame PH: Interpolated Frame 240, 340: Coordinate P list 250: Object region and its initial motion vector 260: Corrected motion vector A, A1~A8, B, B1~B8: Motion vectors coor1, coor2, coor3, coor4: Coordinate candidate set

Claims

1. An image processing method, applicable to an image processing apparatus, comprising: The image processing device receives an I-frame and generates a coordinate I-list, wherein the coordinate I-list corresponds to a first detected object in the I-frame and includes a first coordinate candidate set with four coordinates; the image processing device receives a P-frame following the I-frame and generates a coordinate P-list, wherein the coordinate P-list corresponds to a second detected object in the P-frame and includes a second coordinate candidate set with four coordinates; the image processing device generates at least one object region and its initial motion vector for an interpolated frame between the I-frame and the P-frame based on the coordinate I-list, the coordinate P-list, and a comparison of the number of coordinate candidate sets in the coordinate I-list with the number of coordinate candidate sets in the coordinate P-list; and the image processing device corrects the initial motion vector to generate a corrected motion vector for the at least one object region of the interpolated frame.

2. The image processing method as claimed in claim 1, wherein after generating the coordinate I list and the coordinate P list, the method further comprises: The image processing device performs coordinate processing to obtain at least one correspondence by matching the coordinate I list with the coordinate P list, wherein the at least one correspondence includes a specified correspondence between the first coordinate candidate set of the coordinate I list and the second coordinate candidate set of the coordinate P list.

3. The image processing method as described in claim 2, wherein the step of performing a matching operation between the coordinate I list and the coordinate P list includes: In response to the fact that the number of coordinate candidate sets in the coordinate P list is less than the number of coordinate candidate sets in the coordinate I list, the image processing device searches from the coordinate I list for the coordinate candidate set that has the minimum distance to the second coordinate candidate set in the coordinate P list to form a correspondence. In response to the fact that the number of coordinate candidate sets in the coordinate P list is greater than the number of coordinate candidate sets in the coordinate I list, the image processing device searches from the coordinate P list for a coordinate candidate set that has the minimum distance to the first coordinate candidate set in the coordinate I list to form another correspondence.

4. The image processing method as described in claim 2, wherein the step of generating the at least one object region and its initial motion vector of the interpolated frame between the I-frame and the P-frame based on the coordinate I list and the coordinate P list includes: The image processing device calculates four coordinate motion vectors based on the specified correspondence; The image processing device generates a designated object region in the at least one object region of the interpolated frame based on the first set of candidate coordinates and the four coordinate motion vectors; and the image processing device generates an initial motion vector for the designated object region of the interpolated frame based on the four coordinate motion vectors.

5. The image processing method as described in claim 4, wherein the four coordinate motion vectors include two horizontal motion vectors and two vertical motion vectors, wherein the horizontal component of the initial motion vector is generated based on the two horizontal motion vectors, and wherein the vertical component of the initial motion vector is generated based on the two vertical motion vectors.

6. The image processing method as described in claim 5, wherein the horizontal component of the initial motion vector is the average, minimum, or maximum value of the two horizontal motion vectors, and wherein the vertical component of the initial motion vector is the average, minimum, or maximum value of the two vertical motion vectors.

7. The image processing method of claim 1, wherein the step of correcting the initial motion vector to generate the corrected motion vector of the at least one object region of the interpolated frame includes: The image processing device adds multiple additional motion vectors to the vicinity of the initial motion vector; The image processing device selects the optimal motion vector that minimizes a predetermined cost function from the initial motion vector and the plurality of additional motion vectors as the corrected motion vector.

8. The image processing method as described in claim 7, wherein the initial motion vector is located at the center of the plurality of additional motion vectors.

9. The image processing method as described in claim 8, wherein the plurality of additional motion vectors are further away from the initial motion vector as the interpolated frame gets closer to the P-frame.

Images