Motion estimation and motion compensation method and video processor thereof

Abstract

The present invention discloses a method for motion estimation and motion compensation and a video processor thereof. The method is used in a video processor and includes the following steps: detecting an input frame rate of a series of input frames; calculating a frame count value, the frame count value representing the number of output frame periods between a current input frame and a previous input frame in the series of input frames; calculating a phase step value based on the frame count value, the phase step value being used to generate a phase coefficient, the phase coefficient being used to generate an interpolated frame as an output frame for each output frame period; and generating the interpolated frame using the phase coefficient based on the current input frame and the previous input frame. The steps of calculating the frame count value, calculating the phase step value, and generating the interpolated frame are performed continuously until the input frame rate is successfully detected.

Description

Technical Field

[0001] The present invention relates to a method for a video processor, and in particular to a motion estimation and motion compensation (MEMC) method that can be used in a video processor and a related video processor. Background Art

[0002] Motion Estimation and Motion Compensation (MEMC) is a technique used for frame interpolation, allowing a series of image frames to be displayed at a higher frame rate. For example, if a 30 Hz original video (such as a film) needs to be displayed at 60 Hz, an interpolated frame must be added between every two adjacent input frames of the original video to achieve twice the frame rate.

[0003] The phase of the interpolated frame is calculated based on the input frame rate and the output frame rate. Therefore, the input frame rate and the output frame rate must be fixed to ensure the calculated phase is stable. Traditionally, the front-end device informs the frame rate converter of the input and output frame rates, allowing the frame rate converter to immediately determine the phase step and phase coefficient (which are used for motion compensation). However, in some new applications, such as the command mode of the Mobile Industry Processor Interface (MIPI), the input frame rate is not fixed but changes at any time based on system commands. Furthermore, the front-end device cannot provide the frame rate converter with information about the input frame rate. Therefore, the frame rate converter must dynamically detect the input frame rate to determine the relationship between the input frame rate and the output frame rate, and motion estimation and motion compensation operations cannot begin until the input frame rate is detected. Furthermore, irregular input frame rates are often difficult to detect, resulting in a long detection time for the input frame sequence. During this detection period, motion estimation and motion compensation cannot be performed because the input frame rate and phase for interpolation are not yet available. Therefore, there is a need for improvement in the existing technology. Summary of the Invention

[0004] Therefore, the main object of the present invention is to propose a novel motion estimation and motion compensation (MEMC) method and a related video processor, wherein motion estimation and motion compensation can be performed before the input frame rate is detected, thereby solving the above-mentioned problem.

[0005] One embodiment of the present invention discloses a method for motion estimation and motion compensation for a video processor. The method includes the following steps: detecting an input frame rate of a series of input frames; calculating a frame count value, the frame count value representing the number of output frame periods between a current input frame and a previous input frame in the series of input frames; calculating a phase step value based on the frame count value, the phase step value being used to generate a phase coefficient, the phase coefficient being used to generate an interpolated frame as an output frame for each output frame period; and generating the interpolated frame using the phase coefficient based on the current input frame and the previous input frame. The steps of calculating the frame count value, calculating the phase step value, and generating the interpolated frame are performed continuously until the input frame rate is successfully detected.

[0006] Another embodiment of the present invention discloses a video processor configured to perform the following steps: detecting an input frame rate of a series of input frames; calculating a frame count value representing the number of output frame periods between a current input frame and a previous input frame in the series of input frames; calculating a phase step value based on the frame count value, the phase step value being used to generate a phase coefficient, the phase coefficient being used to generate an interpolated frame as an output frame for each output frame period; and generating the interpolated frame using the phase coefficient based on the current input frame and the previous input frame. The steps of calculating the frame count value, calculating the phase step value, and generating the interpolated frame are performed continuously until the input frame rate is successfully detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 FIG. 1 is a schematic diagram of a display system according to an embodiment of the present invention.

[0008] Figure 2 This is a flowchart of an image processing process according to an embodiment of the present invention.

[0009] Figure 3A and Figure 3B A schematic diagram of a series of output frames and input frames of a video processor.

[0010] The description of the accompanying drawings is as follows:

[0011] 10 Display System

[0012] 12 Video Provider

[0013] 104 Frame Rate Converter

[0014] 106 Video Processor

[0015] 108 display driver

[0016] 110 display screen

[0017] 20 Image Processing Process

[0018] Steps 202-212 DETAILED DESCRIPTION

[0019] Please refer to Figure 1 , Figure 1 FIG. 1 is a schematic diagram of a display system 10 according to an embodiment of the present invention. Figure 1 As shown, the display system 10 may be, for example, a television, which may receive and display an original video having a series of image frames provided by a video providing unit 12. The video providing unit 12 may be, for example, a Digital Versatile Disc (DVD) player or a video streaming service provider, which may communicate with the display system 10 via a wired network or a wireless network. The display system 10 includes a video processor 106, a display driver 108, and a display screen 110. Generally speaking, the frame rate of the original video is different from the frame rate to be displayed on the display screen 110. The video processor 106 may include a television control integrated circuit and a frame rate conversion (FRC) integrated circuit for converting the original video, which generally has a lower frame rate, to generate a series of output frames with a higher frame rate for display through the display screen 110. Alternatively, the video processor 106 may be a television control integrated circuit with a built-in frame rate conversion function, that is, Figure 1 The illustrated frame rate converter 104 can be considered an independent frame rate conversion integrated circuit or a frame rate conversion circuit within a television control integrated circuit. For example, when the frame rate converter 104 receives a 24 Hz input video having two input frames A and B, it can convert the two input frames A and B into five 60 Hz image frames A, A, A, B, B, and perform motion estimation and motion compensation (MEMC) based on the input frames to generate appropriate interpolated frames.

[0020] In another embodiment, the video processor 106 may include a graphics processing unit (GPU) and an independent frame rate conversion integrated circuit, or the video processor 106 may be a graphics processing unit with a built-in frame rate conversion circuit.

[0021] The display driver 108 may convert image data into data voltage signals and use the data voltage signals to drive the display screen 110 to display image frames. The display driver 108 may include a timing controller, a source driver, a gate driver, and / or any other components that may be used to drive the display screen 110. The display screen 110 may be of any type, such as, but not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display panel (PDP).

[0022] Please refer to Figure 2 , Figure 2 FIG. 2 is a flow chart of an image processing process 20 according to an embodiment of the present invention. The image processing process 20 can be implemented in a video processor, such as Figure 1 The video processor 106 in FIG. Figure 2 As shown, the image processing flow 20 includes the following steps:

[0023] Step 202: Receive a series of input frames.

[0024] Step 204: Detect an input frame rate of the series of input frames.

[0025] Step 206: When the input frame rate is detected, calculate the phase step according to the relationship between the input frame rate and the output frame rate.

[0026] Step 208: When the input frame rate is not detected, calculate the phase step according to the frame counting value.

[0027] Step 210: Generate a phase coefficient according to the phase step.

[0028] Step 212: Generate an interpolated frame using the phase coefficient according to the current input frame and the previous input frame.

[0029] According to the image processing flow 20, the video processor first receives a series of input frames (step 202). In one embodiment, the video processor may receive the series of input frames from a front-end device, which may be, for example, Figure 1 The video providing unit 12 in.

[0030] After receiving the input frame, the video processor can detect the input frame rate of the input frame (step 204). Since the front-end device cannot provide frame rate information to the video processor, the video processor needs to detect the input frame rate while receiving the input frame.

[0031] After the video processor detects the input frame rate, it can calculate the phase step based on the relationship between the input frame rate and the output frame rate (step 206). For example, if the input frame rate is 24 Hz and the output frame rate is 60 Hz, it can be determined that the phase step is 2 / 5. Then, the video processor can generate a phase coefficient based on the phase step (step 210). For example, if the phase step is equal to 2 / 5, the phase coefficients are 0, 2 / 5, 4 / 5, 1 / 5 and 3 / 5 in sequence and repeat continuously. Through motion compensation, five consecutive output frames (interpolated frames) can be generated based on two consecutive input frames. Therefore, based on the current input frame and the previous input frame, the video processor can use the phase coefficient to generate an interpolated frame (step 212). In other words, based on the value of the corresponding phase coefficient, the image content of the interpolated frame can be combined with the image content of the previous input frame (considered as a reference frame) to obtain the image content of the interpolated frame. The video processor thus outputs a series of output frames including the interpolated frames.

[0032] As described above, when the input frame rate has not yet been detected, the phase step cannot be determined based on the relationship between the input frame rate and the output frame rate. Therefore, before the input frame rate is detected, the video processor may use another method to calculate the phase step to avoid being unable to perform motion estimation and motion compensation during the long period of time required to detect the input frame rate. According to the image processing process 20, the video processor may calculate the phase step based on the frame count value (step 208). The phase step generated in step 208 serves as a temporary phase step for use during the period when the input frame rate has not yet been detected. The frame count value is a numerical value representing the number of output frame periods between a current input frame and a previous input frame in the series of received input frames that do not include any input frames. In other words, the frame count value indicates the number of output frame periods that do not include any input frames that have elapsed between the reception of the previous input frame and the reception of the current input frame.

[0033] Therefore, the video processor can include two operating modes. In a first mode, the input frame rate is known or detected. Therefore, the video processor can calculate the phase step and phase coefficient based on the relationship between the input and output frame rates, and perform motion estimation and motion compensation based on the calculated phase step and phase coefficient to generate interpolated frames. In a second mode, the input frame rate is unknown or has not yet been detected. The video processor can calculate the phase step and phase coefficient based on the frame count value, and perform motion estimation and motion compensation based on the calculated phase step and phase coefficient to generate interpolated frames. In this case, the video processor can continue to calculate the frame count value, calculate the phase step based on the frame count value, and generate interpolated frames based on the phase coefficient until the input frame rate is successfully detected. Therefore, even if the input frame rate has not yet been detected, the video processor can still perform motion estimation and motion compensation based on the phase step calculated from the frame count value to generate interpolated frames.

[0034] In one embodiment, the video processor may detect whether an input frame is received during each output frame period to determine the frame count value.

[0035] In one embodiment, according to the frame count value, the phase step can be calculated using the following formula:

[0036]

[0037] Where P_Step is the phase step, and FCV is the latest frame count value. The frame count value is continuously calculated and updated during each output frame period, which is the period during which an output frame is generated. For example, when the output frame rate is 60 Hz, the output frame period is equal to 1 / 60 second. For each output frame period, when no input frames are received during that output frame period, the frame count value is incremented by 1. When an input frame is received during that output frame period, the frame count value is used to calculate the phase step, and then the frame count value is reset to 0. Therefore, during an output frame period when an input frame is received, the phase step is calculated using the latest frame count value according to the above formula. In other words, the phase step is calculated by taking the inverse of the sum of the accumulated frame count value and 1.

[0038] Table 1 illustrates an embodiment in which the output frame rate is 60 Hz and the input frame rate is 24 Hz. The video processor knows the output frame rate is 60 Hz, but does not know the input frame rate. Therefore, it must detect the input frame rate by receiving input frames. In this example, input frames A, B, C, D, E, F, ... are received sequentially, and the video processor detects whether an input frame is received during each output frame period. As shown in Table 1, an input reception indication can be used to indicate the reception of an input frame, where a value of "1" indicates that an input frame was received during an output frame period, and a value of "0" indicates that no input frame was received during an output frame period.

[0039]

[0040] Table 1

[0041] Next, the video processor calculates a frame count value based on the input reception indication. As shown in Table 1, during an output frame period when no input frame is received (i.e., the input reception indication is 0), the frame count value is incremented by 1. During an output frame period when an input frame is received (i.e., the input reception indication is 1), the frame count value is reset to 0. Before the frame count value is reset to 0, the most recent frame count value is used to calculate the phase step. For example, if the most recent frame count value is 2, the phase step value is 1 / 3; if the most recent frame count value is 1, the phase step value is 1 / 2. The video processor then generates a phase coefficient based on the phase step value. As shown in Table 1, during the period when the current frame is A and the previous frame is Z, the phase step value is 1 / 2, and the phase coefficients are 0 and 1 / 2. When the phase coefficient reaches 1, it returns to 0 and begins accumulation again, with the current and previous input frames used for interpolation being taken from the next input frame. Next, when the current frame is B and the previous frame is A, the phase step is 1 / 3 and the phase coefficients are 0, 1 / 3 and 2 / 3. When the current frame is C and the previous frame is B, the phase step is 1 / 2 and the phase coefficients are 0 and 1 / 2.

[0042] In this example, when the video processor receives input frame D, it successfully detects the input frame rate. The phase steps and phase coefficients then follow the normal film pattern of 3:2, corresponding to a 24 Hz input frame rate and a 60 Hz output frame rate.

[0043] Therefore, according to an embodiment of the present invention, a method for calculating the number of frames can be used to calculate a temporary phase step that can be used by the video processor until the input frame rate is successfully detected. While the input frame rate is not yet detected and / or determined, this temporary phase step can be used to perform motion estimation and motion compensation to generate interpolated frames. Once the input frame rate is detected and / or determined, a phase step determined by the relationship between the input and output frame rates can be used instead.

[0044] Generally, when a display system is powered on, video display begins before the input frame rate is detected. Therefore, compared to the existing situation in which motion estimation and motion compensation are not performed before the input frame rate is successfully detected, the method of performing motion estimation and motion compensation using the temporal phase step calculated based on the frame count value to generate interpolated frames can still achieve improved image quality.

[0045] Please refer to Figure 3A and 3B , Figure 3A and 3BA video processor (such as Figure 1 Schematic diagram of a series of output frames and input frames of the video processor 106 in FIG. Figure 3A The figure shows the case where the input frame rate is successfully detected, wherein the output frames are generated according to the normal phase steps and phase coefficients determined by the relationship between the input and output frame rates (for example, in film mode 32, the phase coefficients are 0, 2 / 5, 4 / 5, 1 / 5, 3 / 5 and so on). In this embodiment, the output frames can be completely smoothed by motion estimation and motion compensation operations.

[0046] Figure 3B The figure shows the case where the input frame rate has not been detected, wherein the output frame is generated according to the phase step and phase coefficient determined by the frame count value (e.g. 0, 1 / 3, 2 / 3, 0, 1 / 2 are repeated continuously). Figure 3B As shown, although the motion estimation and motion compensation performed according to the temporal phase step cannot achieve completely smooth image movement like the normal phase coefficient, it can still achieve a certain degree of image quality improvement compared to the case where motion estimation and motion compensation are not performed, and can reduce the image content jitter on the output frame.

[0047] It is worth noting that in this example, the video processor is configured to receive unduplicated input frames, typically at an input frame rate of 24 Hz or 30 Hz, and the frame rate converter in the video processor is configured to generate output frames at a higher output frame rate (e.g., 60 Hz or 120 Hz). In this case, the video processor may calculate a frame count value based on an input reception indication, which indicates whether an input frame has been received during each output frame period. In another embodiment, the video processor may receive a series of input frames generated by pull-down duplication, which may be represented as A1, A2, A3, B1, B2, C1, C2, C3, D1, D2, etc., where the input frame rate may be 60 Hz.

[0048] Table 2 shows an embodiment in which the output frame rate is 60 Hz and the input frame rate is 60 Hz, in an application where the input frames received by the video processor (the original input frames in Table 2) are previously copied via pull-down. In this example, since the original input frames are received during each output frame period, the input reception indicator remains "1." Therefore, the video processor further calculates the difference between each two adjacent original input frames (e.g., by accumulating the pixel data differences of all pixels in each two adjacent input frames) and uses the frame difference as a flag to indicate the degree of difference between each two adjacent original input frames.

[0049]

[0050] Table 2

[0051] As shown in Table 2, for example, if the difference between the current raw input frame and the previous raw input frame (e.g., B1 and A3) is greater than a threshold value, the frame difference is "1," meaning that the currently received raw input frame (B1) has new image content. If the difference between the current raw input frame and the previous raw input frame (e.g., A3 and A2) is less than a threshold value, the frame difference is "0," meaning that the currently received raw input frame (A3) is a copy of the previous raw input frame (A2). In the embodiment of Table 2, the input reception indication is replaced by the frame difference in calculating the frame count value. Specifically, if the frame difference is "0," the frame count value is incremented by 1; if the frame difference is "1," the frame count value is reset to 0, and the latest frame count value before the reset is used to calculate the phase step.

[0052] In this example, when the frame difference value indicates that the difference between the current raw input frame and the previous raw input frame is greater than a threshold value, the video processor may select the current raw input frame as one of the frames in the series of input frames (denoted as "input frame" in the second row of Table 2). Therefore, the method for generating the frame count value, phase step, and phase coefficient for the raw input frame in Table 2 is the same as that described in the first embodiment and Table 1. Based on the phase step calculated from the frame count value, the input frame generated by the video processor can be further used to perform motion estimation and motion compensation, as described in the aforementioned paragraphs.

[0053] For example, assume that a video processor receives a series of original input frames A1, A2, A3, B1, B2, C1, C2, C3, D1, D2..., and the original input frames A1, B1, C1, and D1 can be used as input frames with a cadence similar to that in Table 1, which can be used to calculate phase coefficients to perform motion estimation and motion compensation through the operations shown in Table 1.

[0054] It is worth noting that the purpose of the present invention is to propose a new motion estimation and motion compensation method, which can be performed based on the phase step calculated from the frame count value when the input frame rate is not detected. Those skilled in the art can make modifications or changes accordingly, and are not limited to this. For example, in the above embodiment, the output frame rate is 60 Hz, and the motion estimation and motion compensation method can be applied to regular or pull-down input frames received at a frequency of 60 Hz, and can also be applied to irregular or pull-down input frames received at a frequency of 24 Hz. The numerical value of the frame rate is only an example for convenience of explanation in the embodiment. In another embodiment, other frame rates can also be used. For example, the video processor can be set to generate output frames with a frequency of 120 Hz, and the phase step and phase coefficient are generated accordingly to achieve a higher output frame rate.

[0055] In the above-described embodiments of the present invention, the operation of calculating the phase step and phase coefficient based on the frame count value can be performed when the input frame rate has not yet been detected. In fact, the related embodiments can also be applied to various situations where the input frame rate or the relationship between the input and output frame rates is unknown or cannot be determined. For example, a video processor may have several settings that can convert one of a plurality of predetermined input frame rates (e.g., 24 Hz and 30 Hz) to one of a plurality of predetermined output frame rates (e.g., 60 Hz and 120 Hz). When the video processor detects that the actual input frame rate is different from any of the predetermined input frame rates, it may determine that it cannot generate a normal phase step and phase coefficient for the detected input frame rate. The frame count value is then used to generate the phase step and perform motion estimation and motion compensation accordingly. In other words, if the detected input frame rate does not meet any of the video processor's settings and / or the video processor cannot determine an appropriate rule based on the detected input frame rate, the frame count method may be used until a feasible input frame rate is detected again. In another embodiment, the input frame rate may change during display. When the video processor detects the change in the input frame rate, it may switch to an operation mode that uses frame count values ​​to calculate the phase step until the new input frame rate stabilizes and is successfully detected by the video processor.

[0056] In summary, the present invention provides a method for motion estimation and motion compensation and a related video processor. When the input frame rate has not yet been detected, the video processor can perform motion estimation and motion compensation, calculating the phase step based on a frame count value, which represents the number of output frames that have elapsed between the reception of the previous input frame and the reception of the current input frame. According to the method of the present invention, motion estimation and motion compensation can be performed before the input frame rate is successfully detected, resulting in improved image quality compared to when motion estimation and motion compensation are not performed. Furthermore, calculating the phase step based on the frame count value allows for rapid determination of the phase step, thus avoiding the drawback of being unable to perform motion estimation and motion compensation during the extended period of time required to detect the input frame rate.

[0057] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for motion estimation and motion compensation, for use in a video processor, characterized in that: include: detecting an input frame rate of a series of input frames; calculating a frame count value, the frame count value being equal to the number of output frame periods between a current input frame and a previous input frame in the series of input frames that do not include any input frames; Calculating a phase step according to the frame count value, wherein the phase step is used to generate a phase coefficient, and the phase coefficient is used to generate an interpolated frame as an output frame during each output frame period; and The interpolated frame is generated using the phase coefficient according to the current input frame and the previous input frame. The steps of calculating the frame count value, calculating the phase step, and generating the interpolated frame are continuously performed until the input frame rate is successfully detected.

2. The method according to claim 1, wherein The step of calculating the frame count value includes: for each output frame period, When no input frame is received during the output frame period, the frame count value is increased by 1; and When an input frame is received during the output frame period, the frame count value is obtained and then reset to 0.

3. The method according to claim 2, wherein The step of calculating the phase step includes obtaining the inverse of the sum of the accumulation of the frame count value and 1 to calculate the phase step.

4. The method according to claim 1, wherein The series of input frames are received from a front-end device via the video processor.

5. The method according to claim 1, wherein Also includes: receiving a series of raw input frames from a front-end device; determining a plurality of frame difference values, wherein each frame difference value is generated based on every two adjacent original input frames in the series of original input frames; and When the frame difference corresponding to a first original input frame in the series of original input frames indicates that the difference between the first original input frame and a previous original input frame located before the first original input frame is greater than a threshold value, the first original input frame is taken out as one of the series of input frames.

6. A video processor, characterized in that: Used to perform the following steps: detecting an input frame rate of a series of input frames; calculating a frame count value, the frame count value being equal to the number of output frame periods between a current input frame and a previous input frame in the series of input frames that do not include any input frames; Calculating a phase step according to the frame count value, wherein the phase step is used to generate a phase coefficient, and the phase coefficient is used to generate an interpolated frame as an output frame during each output frame period; and The interpolated frame is generated using the phase coefficient according to the current input frame and the previous input frame. The steps of calculating the frame count value, calculating the phase step, and generating the interpolated frame are continuously performed until the input frame rate is successfully detected.

7. The video processor according to claim 6, wherein: The steps for calculating the frame count value include: For each output frame period, When no input frame is received during the output frame period, the frame count value is increased by 1; and When an input frame is received during the output frame period, the frame count value is obtained and then reset to 0.

8. The video processor according to claim 7, wherein: The steps to calculate the phase step are: The phase step is calculated by obtaining the inverse of the sum of the accumulated frame count value and 1.

9. The video processor according to claim 6, wherein: The series of input frames are received from a front-end device via the video processor.

10. The video processor according to claim 6, wherein: The method is further configured to perform the following steps: receiving a series of original input frames from a front-end device; determining a plurality of frame difference values, wherein each frame difference value is generated based on every two adjacent original input frames in the series of original input frames; and When the frame difference corresponding to a first original input frame in the series of original input frames indicates that the difference between the first original input frame and a previous original input frame located before the first original input frame is greater than a threshold value, the first original input frame is taken out as one of the series of input frames.

Images