Motion estimation and motion compensation method and video processor thereof

Abstract

The present invention discloses a motion estimation and motion compensation method for a video processor and the video processor thereof. The motion estimation method includes the following steps: calculating a frame difference between a current input frame and a previous input frame; comparing the frame difference with a critical value to generate a comparison result; if the comparison result indicates that the frame difference is less than the critical value, determining whether the first block is a mixed block based on a block difference indicating the difference between a first block on the current input frame and a block located at the same position on the previous input frame; generating a temporal motion vector candidate for the first block; when the first block is determined to be a mixed block, modifying the temporal motion vector candidate based on the current position difference of a video portion of the current input frame; and determining a motion vector for the first block from a plurality of motion vector candidates including the temporal motion vector candidate.

Description

Technical Field

[0001] The present invention relates to a motion estimation and motion compensation (MEMC) method and a related video processor, and in particular to a motion estimation and motion compensation method with mixed image content that can be used in a video processor. Background Art

[0002] Motion Estimation and Motion Compensation (MEMC) is a technique used for frame interpolation, allowing a series of image frames to be displayed at a higher frame rate. For example, if a 30 Hz original video (such as a film) needs to be displayed at 60 Hz, an interpolated frame is added between every two consecutive input frames of the original video to achieve twice the frame rate.

[0003] Assuming the original video from the video provider has a 24 Hz frame rate and the image to be displayed on the display is 60 Hz, frame conversion is used to convert every two input frames into five output frames. To avoid judder in the output image, interpolation is performed using the cadence of film mode 32, where two input frames are combined to generate at least one interpolated frame. Motion estimation and motion compensation techniques enable the video processor to select the appropriate input frame and perform interpolation using the correct phase coefficients, resulting in a smooth output image.

[0004] However, if the video providing unit provides a mixed-mode image sequence composed of a film portion and a video portion, such as a scrolling text image at the bottom or side of the screen, or a text message popping up above the film portion, conventional motion estimation and motion compensation operations often cannot process both the video and film portions simultaneously. Generally speaking, motion estimation and motion compensation can only process the video portion, resulting in jitter in the film portion. Therefore, there is a need for improvement in the existing technology. Summary of the Invention

[0005] Therefore, the main purpose of the present invention is to propose a novel motion estimation and motion compensation (MEMC) method and a related video processor, which can be used to process mixed-mode images to solve the above-mentioned problems.

[0006] One embodiment of the present invention discloses a motion estimation method for a video processor. The motion estimation method includes the following steps: receiving a current input frame, the current input frame including a film part and a video part; calculating a frame difference between the current input frame and a previous input frame located before the current input frame; comparing the frame difference with a first threshold to generate a comparison result; when the comparison result indicates that the frame difference is less than the first threshold, determining whether a first block among a plurality of blocks on the current input frame is a mixed block based on a block difference value for the first block, wherein the block difference value indicates a difference between the first block on the current input frame and a block on the previous input frame located at the same position as the first block; generating at least one temporal motion vector candidate for the first block. candidate); when the first block is determined to be the mixed block, modifying the at least one temporal motion vector candidate for the first block according to a current position difference of the video portion of the current input frame; and determining a motion vector for the first block from a plurality of motion vector candidates, the plurality of motion vector candidates including the at least one temporal motion vector candidate.

[0007] Another embodiment of the present invention discloses a video processor that can be used to perform the following steps: receiving a current input frame, the current input frame including a film portion and a video portion; calculating a frame difference between the current input frame and a previous input frame located before the current input frame; comparing the frame difference with a first threshold to generate a comparison result; when the comparison result indicates that the frame difference is less than the first threshold, determining whether a first block among a plurality of blocks on the current input frame is a mixed block based on a block difference with respect to the first block, wherein the block A method for determining a motion vector for the first block includes providing a block difference value indicating a difference between the first block on the current input frame and a block on the previous input frame that is co-located with the first block; generating at least one temporal motion vector candidate for the first block; modifying the at least one temporal motion vector candidate for the first block according to a current position difference value of the video portion of the current input frame when the first block is determined to be the mixed block; and determining a motion vector for the first block from a plurality of motion vector candidates, the plurality of motion vector candidates including the at least one temporal motion vector candidate.

[0008] Another embodiment of the present invention discloses a motion compensation method for a video processor. The motion compensation method includes the following steps: receiving a current input frame, the current input frame including a film portion and a video portion; judging whether the first block is a mixed block based on a block difference value about a first block among a plurality of blocks on the current input frame, wherein the block difference value indicates the difference between the first block on the current input frame and a block on a previous input frame that is located at the same position as the first block; when the first block is judged to be the mixed block, generating a phase coefficient for the first block based on a ratio of a first position difference value to a second position difference value, wherein the The first position difference is a difference between a position of a video portion of an interpolated frame and a position of a video portion of the previous input frame, and the second position difference is a difference between a position of the video portion of the current input frame and a position of the video portion of the previous input frame; and the interpolated frame is generated using the phase coefficient based on the current input frame and the previous input frame, wherein a block on the interpolated frame is generated using the phase coefficient based on the first block on the current input frame and a second block on the previous input frame indicated by a motion vector for the first block of the current input frame.

[0009] Another embodiment of the present invention discloses a video processor that can be used to perform the following steps: receiving a current input frame, the current input frame including a film portion and a video portion; determining whether a first block is a mixed block based on a block difference value for a first block among a plurality of blocks on the current input frame, wherein the block difference value indicates a difference between the first block on the current input frame and a block on a previous input frame that is located at the same position as the first block; when the first block is determined to be the mixed block, generating a position difference value for the first block based on a ratio of a first position difference value to a second position difference value. a phase coefficient, wherein the first position difference is the difference between the position of a video portion of an interpolated frame and the position of a video portion of the previous input frame, and the second position difference is the difference between the position of the video portion of the current input frame and the position of the video portion of the previous input frame; and generating the interpolated frame using the phase coefficient based on the current input frame and the previous input frame, wherein a block on the interpolated frame is generated using the phase coefficient based on the first block on the current input frame and a second block on the previous input frame indicated by a motion vector for the first block of the current input frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 FIG. 1 is a schematic diagram of a display system according to an embodiment of the present invention.

[0011] Figure 2 Show Figure 1A detailed implementation of the video processor in.

[0012] Figure 3 This is a flowchart of an image processing process according to an embodiment of the present invention.

[0013] Figure 4 FIG. 1 is a schematic diagram of a mixed mode image frame including a film portion and a video portion.

[0014] Figure 5 FIG. 1 is a schematic diagram illustrating mapping block differences to block mixing levels according to an embodiment of the present invention.

[0015] Figure 6 This is a flowchart of another image processing process according to an embodiment of the present invention.

[0016] Figure 7 Schematic diagram of combining input frames to generate interpolated frames using normal phase coefficients and motion of a film portion.

[0017] Figure 8 Schematic diagram of combining input frames to generate interpolated frames using normal phase coefficients and motion of video parts.

[0018] Figure 9 Schematic diagram of combining input frames to generate interpolated frames using mixed phase coefficients and motion of video parts.

[0019] The description of the accompanying drawings is as follows:

[0020] 10 Display System

[0021] 12 Video Provider

[0022] 104 Frame Rate Converter

[0023] 106 Video Processor

[0024] 108 display driver

[0025] 110 display screen

[0026] 202 Mixed Block Detector

[0027] 204 Motion Estimation Unit

[0028] 206 Motion Compensation Unit

[0029] 30, 60 Image processing process

[0030] Steps 300-316, 600-610

[0031] TH threshold

[0032] A1, A2, B1, B2 input frames DETAILED DESCRIPTION

[0033] The video processor of the present invention receives a series of input frames of raw video from a video providing unit, performs motion estimation and motion compensation (MEMC) on these input frames to generate interpolated frames, and then outputs a series of output frames including the interpolated frames. As described above, each image frame received from the video providing unit in hybrid mode may include a video portion and a film portion. The film portion represents a background image, while the video portion may be a scrolling text image, pop-up text, a dialog box, or any other image object overlaid on the film. Assuming the raw video has a frame rate of 24 Hz and is required to be converted to a 60 Hz frame rate, a 3:2 pull-down process may be used to generate the film portion of the series of input frames. In the series of input frames, the image content of the film portion may be represented as A, A, A, B, B, C, C, C, D, D, ..., where three input frames A are similar to each other, two input frames B are similar to each other, and so on.

[0034] In addition, in blending mode, the input frames are further blended with the video portion. Generally speaking, a series of input frames are pulled down 3:2 before the video portion is added. This video portion is typically generated at a 60 Hz frame rate. Therefore, the image content of the video portion of each input frame is different, and can be represented as a, b, c, d, e, etc.

[0035] Please refer to Figure 1 , Figure 1 FIG. 1 is a schematic diagram of a display system 10 according to an embodiment of the present invention. Figure 1 As shown, the display system 10 may be, for example, a television, which may receive and display an original video having a series of image frames provided by a video providing unit 12. The video providing unit 12 may be, for example, a Digital Versatile Disc (DVD) player or a video streaming service provider, which may communicate with the display system 10 via a wired or wireless network. The display system 10 includes a video processor 106, a display driver 108, and a display screen 110. Generally speaking, the frame rate of the original video is different from the frame rate to be displayed on the display screen 110. The video processor 106 may include a frame rate conversion (FRC) integrated circuit for converting the original video with a lower frame rate to generate a series of output frames with a higher frame rate for display through the display screen 110. Alternatively, the video processor 106 may be a television control integrated circuit with a frame rate conversion function, that is, Figure 1The illustrated frame rate converter 104 can be considered an independent frame rate conversion integrated circuit or a frame rate conversion circuit within a television control integrated circuit. For example, when the frame rate converter 104 receives a 24 Hz input video having two input frames A and B, it can convert the two input frames A and B into five 60 Hz image frames A, A, A, B, B. These image frames can be combined with the video portion as a film portion, and the video processor 106 can perform motion estimation and motion compensation operations based on these image frames to generate appropriate interpolated frames.

[0036] The display driver 108 can convert image data into data voltage signals and use the data voltage signals to drive the display screen 110 to display image frames. The display driver 108 may include a timing controller, a source driver, a gate driver, and / or any other components that can be used to drive the display screen 110. The display screen 110 can be of any type, such as, but not limited to, a liquid crystal display (LCD), a light-emitting diode (LED), and a plasma display panel (PDP).

[0037] The video processor 106 may be configured to perform motion estimation and motion compensation to generate interpolated frames, thereby making the output frames smoother. Figure 2 FIG. 1 shows a detailed implementation of the video processor 106. Figure 2 As shown, the video processor 106 includes a mixed block detector 202, a motion estimation unit 204, and a motion compensation unit 206. The mixed block detector 202, motion estimation unit 204, and motion compensation unit 206 are part of the frame rate converter 104, which can be a separate integrated circuit or a built-in circuit.

[0038] Motion estimation and motion compensation are typically block-based operations. To efficiently process mixed-mode image frames, the video processor 106 must determine whether each image block is a mixed block containing video content. Specifically, the mixed block detector 202 determines whether each image block is a mixed block and outputs the determination result to the motion estimation unit 204 and the motion compensation unit 206. During the motion estimation process, the motion estimation unit 204 must identify the best motion vector (MV) for each block in the input frame from a plurality of motion vector (MV) candidates. These MV candidates may include, but are not limited to, one or more temporal MV candidates and / or one or more spatial MV candidates. If a block is determined to be a mixed block, the motion estimation unit 204 may apply a gain to the temporal MV candidate for the mixed block, modify the temporal MV candidate based on the gain, and then select the best MV from the MV candidates for the mixed block.

[0039] During the motion compensation process, the motion compensation unit 206 generates interpolated frames based on the motion vectors selected from each block, using a phase coefficient to generate each interpolated frame. Generally, the normal phase coefficient used for interpolation is determined based on the cadence of the input frame and the input / output frame rate, with uniform phase steps. For example, if the output frame rate is the same as the input frame rate and the cadence is film mode 32, the normal phase coefficients are 0 (i.e., 1*A+0*B), 2 / 5 (i.e., (3 / 5)*A+(2 / 5)*B), 4 / 5 (i.e., (1 / 5)*A+(4 / 5)*B), 1 / 5 (i.e., (4 / 5)*B+(1 / 5)*C), and 3 / 5 (i.e., (2 / 5)*B+(3 / 5)*C). If an image block is determined to be a mixed block, the motion compensation unit 206 may generate an interpolated frame using mixed phase coefficients different from normal phase coefficients, wherein one or more mixed phase coefficients are different from the corresponding normal phase coefficients.

[0040] On the contrary, if an image block is determined not to be a mixed block, the motion estimation unit 204 does not modify the temporal motion vector candidate of the image block, and the motion compensation unit 206 uses normal phase coefficients with uniform phase steps to generate an interpolated frame.

[0041] Please refer to Figure 3 , Figure 3 FIG. 3 is a flow chart of an image processing process 30 according to an embodiment of the present invention. The image processing process 30 can be implemented in a video processor, such as Figure 1 and Figure 2 The video processor 106 in FIG. Figure 3 As shown, the image processing flow 30 includes the following steps:

[0042] Step 300: Start.

[0043] Step 302: Receive a current input frame, the current input frame including a film portion and a video portion.

[0044] Step 304: Calculate a frame difference between the current input frame and a previous input frame located before the current input frame.

[0045] Step 306: Compare the frame difference value with a first threshold to generate a comparison result.

[0046] Step 308 : When the comparison result indicates that the frame difference is less than the first threshold, determine whether the first block is a mixed block according to a block difference associated with a first block among the plurality of blocks on the current input frame.

[0047] Step 310: Generate at least one temporal motion vector candidate for the first block.

[0048] Step 312: When the first block is determined to be a mixed block, modify at least one temporal motion vector candidate for the first block according to a current position difference of the video portion of the current input frame.

[0049] Step 314: Determine a motion vector for the first block from a plurality of motion vector candidates, the plurality of motion vector candidates including at least a temporal motion vector candidate.

[0050] Step 316: End.

[0051] According to the image processing flow 30, the video processor 106 may receive a current input frame having a film portion and a video portion (step 302), and receive a previous input frame preceding the current input frame, where the previous input frame may also have a film portion and a video portion. In one embodiment, the video processor 106 may include a frame buffer for storing image data, or read image data of the input frame from an external frame buffer. As described above, the content of the film portion, for example, may be a background image, which may follow a regular film pattern. The content of the video portion, for example, may be a scrolling text image overlaid on the film, which may be different in each image frame. For example, the text in the scrolling text image may continuously move, resulting in a displacement in each input frame compared to the previous input frame. To achieve optimal image quality and avoid unnecessary judder, the motion estimation and motion compensation processes must consider both the video portion and the film portion.

[0052] After the video processor 106 receives the current input frame, it calculates the frame difference between the current input frame and the previous input frame (step 304), and compares the frame difference with a threshold to generate a comparison result (step 306). Note that the frame difference indicates the difference between the entire image frame and can be used to determine whether the current input frame and the previous input frame belong to the same frame group. Generally speaking, similar input frames can be classified into the same frame group. For example, for the five consecutive input frames A, A, A, B, and B generated after 3:2 pull-down, the three input frames A can be classified into one frame group, and the two input frames B can be classified into another frame group. Therefore, if the comparison result indicates that the frame difference is greater than the threshold, it can be determined that the current input frame and the previous input frame are in different frame groups; if the comparison result indicates that the frame difference is less than the threshold, it can be determined that the current input frame and the previous input frame are in the same frame group.

[0053] Please refer to Figure 4 , Figure 4 Schematic diagram of a mixed mode image frame including a film portion and a video portion. Figure 4 As shown, the film part refers to the picture, and the video part refers to the scrolling text image located below the image frame. Figure 4 A bit sequence (0 or 1) is also shown, representing whether the film portion or video portion of each current frame is sufficiently different from the film portion or video portion of the previous frame, similar to the comparison result sequence generated in step 306 . Figure 4 Assuming that the film portion of a series of input frames is received according to film mode 32, the film content can be represented as A, A, A, B, B, C, C, C..., so the corresponding bit sequence is 1, 0, 0, 1, 0, 1, 0, 0..., and so on. Bit "1" represents that the film portion or video portion of the current frame is significantly different from the film portion or video portion of the previous frame (i.e., the difference is large enough), and bit "0" represents that the film portion or video portion of the current frame is not significantly different from the film portion or video portion of the previous frame (i.e., the difference is not large enough). In the same series of input frames, the video portion between each two consecutive input frames is often different, so the corresponding sequence of the video portion is 1, 1, 1, 1, 1. It should be noted that in the image processing process 30, the frame difference and comparison results are calculated on a frame-based basis, rather than only based on the film portion or video portion.

[0054] In an embodiment of the present invention, motion estimation and motion compensation are performed differently for blocks in the video portion and blocks in the film portion. In short, if the frame difference of a current frame is determined to be sufficiently small (e.g., less than a threshold), the video processor 106 can determine whether each block in the current frame belongs to the video portion (referred to as a mixed block) based on the block difference. For blocks in the video portion of the current input frame, the video processor 106 performs motion estimation and motion compensation in a special manner.

[0055] Generally speaking, the film portion of an image frame occupies a larger area than the video portion. Therefore, the overall frame difference of the image frame is primarily determined by the frame difference of the film portion. Therefore, an appropriate threshold can be set so that the video processor 106 can determine whether each input frame has a sufficiently small frame difference. Subsequent mixed block detection is performed only on input frames that have been determined to have a sufficiently small frame difference.

[0056] In one embodiment, a group change flag may be used on each input frame to indicate the group setting. For example, if a current input frame is assigned to a different frame group than the previous input frame, the group change flag of the current input frame may be set to "1." If a current input frame is assigned to the same frame group as the previous input frame, the group change flag of the current input frame may be set to "0." In this case, mixed blocks may be detected based on the group change flag. More specifically, mixed blocks may be detected for input frames whose group change flag value is "0," and the comparison result generated in step 306 may be used to set the value of the group change flag.

[0057] Therefore, when the frame difference comparison result indicates that the frame difference value is less than the threshold value, the video processor 106 will further determine whether each block is a mixed block based on the block difference value of each block. In other words, when the video processor 106 determines that the frame difference value is less than the threshold value, it can be determined that the video portion of the current input frame is similar to the video portion of the previous input frame, and thus determine which blocks belong to the video portion.

[0058] In one embodiment, the mixed block detector 202 of the video processor 106 may determine whether a first block is a mixed block based on a block difference value associated with the first block in the current input frame (step 308). The block difference value indicates the difference between the first block in the current input frame and a block located at the same position as the first block in the previous input frame. It should be noted that the frame difference value represents the difference across the entire image frame, while the block difference value represents the difference between a block at a specific position between different image frames.

[0059] Specifically, the mixed block detector 202 may sum the differences between each pixel data item in a first block in the current input frame and each corresponding pixel data item in a corresponding block in a previous input frame to calculate a block difference value for the first block. The corresponding block in the previous input frame is located at the same position as the first block in the current input frame, and the differences in each pixel data item may be summed to calculate the block difference value. The mixed block detector 202 thus determines the degree of block mixing for the first block based on the block difference value.

[0060] Please refer to Figure 5 , Figure 5 FIG. 1 is a schematic diagram of mapping block differences to block mixing levels according to an embodiment of the present invention. Figure 5 As shown, the block mixing degree is determined based on the block difference. If the block difference of the image frame is larger, the block mixing degree has a larger value; if the block difference of the image frame is smaller, the block mixing degree has a smaller value. The mixed block detector 202 also compares the block mixing degree of the first block with a threshold value TH to determine whether the first block is a mixed block. In detail, if the block mixing degree is greater than the threshold value TH, the first block can be determined to be a mixed block; if the block mixing degree is less than the threshold value TH, the first block can be determined to be a non-mixed block. For non-mixed blocks, the video processor 106 (or the frame rate converter 104) performs motion estimation and motion compensation operations in a different manner than the motion estimation and motion compensation operations performed for mixed blocks.

[0061] It is worth noting that the threshold TH can be set flexibly. The threshold TH can be any level that can be used to distinguish between mixed blocks mixed with the video portion and non-mixed blocks with only the video portion. Different motion estimation and motion compensation methods can be used for mixed blocks and non-mixed blocks.

[0062] After determining whether the first block is a mixed block based on the block difference, the mixed block detector 202 may output the determination result to the motion estimation unit 204 and the motion compensation unit 206. Therefore, based on whether the first block is a mixed block, the motion estimation unit 204 may perform motion estimation on the first block using an appropriate method, and the motion compensation unit 206 may perform motion compensation on the first block using an appropriate method.

[0063] Specifically, the motion estimation unit 204 may generate at least one temporal motion vector candidate for the first block (step 310). For the first block in the current input frame, the motion estimation unit 204 may generate one or more motion vector candidates based on the block motion of the first block. If the first block uses the best known motion vector of another block in the current input frame as a motion vector candidate, the motion vector candidate is a spatial motion vector candidate. If the block motion of the first block is referenced to a block in a previous input frame, the generated motion vector candidate is a temporal motion vector candidate. In the present invention, if the first block is determined to be a mixed block, the motion estimation unit 204 may further modify the at least one temporal motion vector candidate for the first block based on a current position difference of the video portion of the current input frame (step 312).

[0064] In one embodiment, the temporal motion vector candidate may be multiplied by a gain determined based on a current position difference of the video portion. More specifically, the motion estimation unit 204 may modify the temporal motion vector candidate by multiplying each temporal motion vector candidate by a gain determined based on a current position difference and a previous position difference of the video portion. The current position difference is the difference between the position of the video portion of the current input frame corresponding to a current output frame and the position of the video portion of the previous input frame, and the previous position difference is the difference between the position of the video portion of the current input frame corresponding to a previous output frame before the current output frame and the position of the video portion of the previous input frame.

[0065] Assume that the video processor 106 receives an image sequence in film mode 22. The film portion of the image sequence may include a series of input frames A1, A2, B1, B2, C1, C2..., wherein input frames A1 and A2 belong to the same frame group and have similar image content, input frames B1 and B2 belong to the same frame group and have similar image content, and so on. The video portion may be a scrolling text image that has an equal displacement between each input frame, that is, the video portion moves at a constant speed. For ease of explanation, the position of the video portion is represented by a number. For example, the image content of a series of input frames may be represented as A1(0), A2(10), B1(20), B2(30)..., which includes information about the film portion and position information (represented by a numerical value) of the video portion.

[0066] In this example, each output frame can be generated by combining an input frame (as the current input frame) and another input frame (as the previous input frame) using a phase coefficient, as shown in Table 1:

[0067]

[0068]

[0069] Table 1

[0070] Table 1 shows how the previous input frame and the current input frame are combined based on phase coefficients to generate a series of output frames (or interpolated frames). The "Previous Video Position" field represents the position of the video portion of the previous input frame, and the "Current Video Position" field represents the position of the video portion of the current input frame. The position difference is the difference between the current video position and the previous video position, indicating the displacement or relative distance of the video portion between the selected current and previous input frames. In this example, the output frame rate is twice the input frame rate. Therefore, each input frame (serving as the current input frame) can be interpolated to generate two output frames based on the appropriate phase coefficient arrangement. The previous input frame is typically selected from the last input frame of the previous frame group. For example, for current input frames B1 and B2, the previous input frame is the last input frame A2 of frame group A; for current input frames C1 and C2, the previous input frame is the last input frame B2 of frame group B.

[0071] As shown in Table 1, under the aforementioned rules for obtaining the previous and current input frames, the relative distance between the selected video portion of the current and previous input frames is not constant. For example, the calculated position differences are 10, 10, 20, 20, 10, 10, 20, 20, ... This is because the rules for obtaining the input frames are based only on the video portion, while the video portion shifts between each input frame. Inconsistent position differences can lead to inaccurate temporal motion vector candidates for the video portion.

[0072] To address this issue, when a block is determined to be a mixed block including a video portion, the motion estimation unit 204 may multiply the temporal motion vector candidate by a gain determined based on the position difference. In one embodiment, this gain is equal to the ratio of the current position difference to the previous position difference. For example, as shown in Table 1, the first output frame is generated using input frame B1 as the current input frame and input frame A2 as the previous input frame, with a position difference of 10. The second output frame is generated using input frame B1 as the current input frame and input frame A2 as the previous input frame, with a position difference of 10. Therefore, the gain applied to the temporal motion vector candidate for the mixed block of the current input frame is 1, which is the ratio of the current position difference corresponding to the second output frame to the previous position difference corresponding to the first output frame. Furthermore, the third output frame is generated using input frame B2 as the current input frame and input frame A2 as the previous input frame, with a position difference of 20. Therefore, the gain applied to the temporal motion vector candidate for the mixed block of the current input frame is 2, which is the ratio of the current position difference corresponding to the third output frame to the previous position difference corresponding to the second output frame. In this case, during the motion estimation operation, the temporal motion vector candidate for the mixed block must be multiplied by 2 to correctly represent the motion of the video portion between the previous input frame and the current input frame.

[0073] The modified temporal motion vector candidate is then used for motion vector selection. The motion estimation unit 204 selects the best motion vector for the mixed block from a plurality of motion vector candidates, including a temporal motion vector candidate (step 314). The motion vector candidates may also include one or more spatial motion vector candidates and / or any other type of motion vector candidate. Of the motion vector candidates, only the temporal motion vector candidate is modified based on the position difference. The selected best motion vector is provided to the motion compensation unit 206 for motion compensation.

[0074] Please refer to Figure 6 , Figure 6 FIG. 6 is a flow chart of another image processing process 60 according to an embodiment of the present invention. The image processing process 60 can be implemented in a video processor, such as Figure 1 and Figure 2 The video processor 106 in FIG. Figure 6 As shown, the image processing process 60 includes the following steps:

[0075] Step 600: Start.

[0076] Step 602: Receive a current input frame, the current input frame including a film portion and a video portion.

[0077] Step 604 : Determine whether a first block is a mixed block according to a block difference value associated with a first block among a plurality of blocks in the current input frame.

[0078] Step 606: When the first block is determined to be a mixed block, generate a phase coefficient for the first block according to a ratio of a first position difference to a second position difference.

[0079] Step 608: Generate an interpolated frame using phase coefficients based on the current input frame and the previous input frame, wherein a block on the interpolated frame is generated using the phase coefficients based on a first block on the current input frame and a second block on the previous input frame indicated by a motion vector for the first block of the current input frame.

[0080] Step 610: End.

[0081] According to the image processing flow 60, the video processor 106 may receive a current input frame having a film portion and a video portion (step 602), and receive a previous input frame located before the current input frame, wherein the previous input frame may also have a film portion and a video portion. After the video processor 106 determines that the current input frame has a sufficiently small frame difference, it may further determine whether each block on the current input frame is a mixed block based on the block difference. In one embodiment, the mixed block detector 202 of the video processor 106 may determine whether the first block is a mixed block based on the block difference of a first block on the current input frame (step 604). The block difference is used to indicate the difference between the first block on the current input frame and the corresponding block on the previous input frame that is located at the same position as the first block. The calculation of the block difference can be achieved by summing the difference between each pixel data of the first block on the current input frame and each pixel data of the corresponding block at the same position on the previous input frame. The relevant operation method can be referred to the aforementioned paragraphs and Figure 5 The description is not repeated here.

[0082] When the first block is determined to be a mixed block, the motion compensation unit 206 of the video processor 106 may generate a phase coefficient for the first block based on a ratio of a first position difference to a second position difference (step 606), where the first and second position differences are determined based on the positions of the video portions. In this example, the current input frame and the previous input frame are combined to generate an interpolated frame. The first position difference is the difference between the position of the video portion of the interpolated frame and the position of the video portion of the previous input frame, and the second position difference is the difference between the position of the video portion of the current input frame and the position of the video portion of the previous input frame.

[0083] For a second block determined to be a non-mixed block that does not include a video portion or includes only a small amount of a video portion, the phase coefficient can be calculated using a normal phase coefficient table according to the regularity of the input frame and the input / output frame rate, and has an equal phase step.

[0084] After the motion compensation unit 206 determines the phase coefficients, it can further utilize the phase coefficients to generate an interpolated frame based on the current input frame and the previous input frame. More specifically, each block in the interpolated frame is generated using the phase coefficients based on a block (e.g., the first block) in the current input frame and a corresponding block in the previous input frame indicated by the motion vector of the block in the current input frame (step 608).

[0085] Please refer to Figure 7 , Figure 7 Schematic diagram of combining input frames to generate interpolated frames using normal phase coefficients and motion of film parts. This example uses film mode 22 and the output frame rate is twice the input frame rate. Therefore, each input frame can be used to generate 2 output frames (or interpolated frames), and the normal phase coefficients are 0, 0.25, 0.5, 0.75, which has a fixed phase step of 0.25. For example, the current input frame B1 and the previous input frame A2 are combined to generate two interpolated frames with phase coefficients of 0 and 0.25 respectively. The current input frame B2 and the previous input frame A2 are combined to generate two interpolated frames with phase coefficients of 0.5 and 0.75 respectively. As shown in FIG. Figure 7 As shown, the video parts of input frames A1 and A2 have similar images, so an object or image of the video parts of input frames A1 and A2 is located at the same position. Similarly, the video parts of input frames B1 and B2 have similar images, so an object or image of the video parts of input frames B1 and B2 is located at the same position. Interpolation performed using normal phase coefficients can produce an image without any jitter in the video part, such as Figure 7 shown.

[0086] However, if the normal phase coefficient is used for the video portion, jitter may occur. Figure 8 , Figure 8 Schematic diagram of combining input frames to generate interpolated frames using normal phase coefficients and motion of video parts. Figure 8 As shown, the position of a video portion varies across each input frame. An object in the video portion moves in the same direction at a constant speed, resulting in the same displacement between input frames A2 and B1 as between input frames B1 and B2. In this case, using normal phase coefficients of 0, 0.25, 0.5, and 0.75 will cause the video portion to appear jittery. In other words, interpolating based on normal phase coefficients will result in inconsistent video portion movement speed across the interpolated frames.

[0087] To solve this problem, the motion compensation unit 206 may calculate a mixed phase coefficient for the mixed block including the video portion. The mixed phase coefficient is generated based on the position difference. An example of a method for generating the mixed phase coefficient is shown in Table 2, as shown below:

[0088]

[0089]

[0090] Table 2

[0091] Table 2 uses the same scenario as Table 1, assuming video mode 22 and an output frame rate twice the input frame rate. Table 2 defines the position of the video portion of the previous input frame in the "Previous Video Position" field and the position of the video portion of the current input frame in the "Current Video Position" field. Furthermore, the "Interpolated Video Position" field represents the position of the video portion of the interpolated frame. The blending phase can be determined based on the current video position, the previous video position, and the interpolated video position.

[0092] To determine the position of the video portion of the interpolated frames, the motion compensation unit 206 of the video processor 106 may calculate a video step. The video step represents the movement or displacement of the video portion of each interpolated frame. The video step may be a constant value, indicating that the video portion (e.g., scrolling text) moves at a constant speed. Specifically, the video step is calculated by dividing a position difference value by the number of interpolated frames. The position difference value is the difference between the position of the video portion of the last frame of the current frame group to which the current input frame belongs and the position of the video portion of a frame in the previous frame group preceding the current frame group. The number of interpolated frames is determined based on the output frame rate and the input frame rate of the video processor 106.

[0093] In this example, each input frame (the current input frame) can generate two output frames through interpolation, and film mode 22 indicates that each frame group has two input frames. Therefore, one frame group image can be used to generate four interpolated frames, making the number of interpolated frames four. When frame groups A and B are combined to generate interpolated frames, the position of the video portion of the last frame of the current frame group is the video position of input frame B2, which is compared with the position of the video portion of a frame in the previous frame group A to determine a position difference. Since the previous input frame is usually selected from the last frame of the previous frame group (i.e., input frame A2), the position of the video portion of the frame in the previous frame group A is the video position of input frame A2. Therefore, the position difference is equal to 20, which is the video position 30 of input frame B2 minus the video position 10 of input frame A2. Therefore, since the position difference is equal to 20 and the number of interpolated frames is four, the video step is 5.

[0094] In this case, the position of the video portion of the current interpolated frame can be obtained by adding the video step 5 to the position of the video portion of the previous frame. This previous frame can be the previous input frame or the previous interpolated frame. As shown in Table 2, the video positions of a series of interpolated frames are 10, 15, 20, 25, 30, 35, 40, 45, etc., and their video step is equal to 5.

[0095] Next, the motion compensation unit 206 calculates a blending phase coefficient based on the current video position, the previous video position, and the interpolated video position. Specifically, the blending phase coefficient is equal to the ratio of the interpolated position difference to the current position difference, where the interpolated position difference is the difference between the position of the video portion of the interpolated frame and the position of the video portion of the previous input frame, and the current position difference is the difference between the position of the video portion of the current input frame and the position of the video portion of the previous input frame. Unlike normal phase coefficients, the blending phase coefficient is used for blended blocks determined to include video portions.

[0096] For example, in Table 2, the interpolated frame in row 2 is generated using input frame B1 as the current input frame and input frame A2 as the previous input frame, with a current position difference of 10 and an interpolated position difference of 5. Therefore, the mixing phase coefficient used for the interpolated frame is 0.5, which is different from the normal phase coefficient of 0.25 obtained according to the rule. In a similar manner, the mixing phase coefficients used for a series of interpolated frames are 0, 0.5, 0.5 and 0.75. The mixing phase coefficients can be used by the mixing block to perform interpolation to produce a jitter-free image in the video portion. Figure 9 As shown, by using mixed phase coefficients, the video portion of a series of interpolated frames can show smooth movement.

[0097] It is worth noting that the purpose of the present invention is to provide a motion estimation and motion compensation method that can be used for mixed-mode image frames having a video portion and a film portion. Those skilled in the art can modify or change the present invention accordingly, and are not limited to this. For example, the above embodiment uses film mode 22 as an example for ease of explanation. In another embodiment, the motion estimation and motion compensation method of the present invention can also be applied to another film mode (such as 32) and to different output frame rates. As long as the video processor can determine whether an image block is a mixed block including image content of a video portion and perform motion estimation and / or motion compensation for the mixed block and the non-mixed block according to different methods, the related motion estimation and / or motion compensation operations should fall within the scope of the present invention.

[0098] In summary, the present invention proposes a novel motion estimation and motion compensation method and video processor for mixed-mode image frames, wherein different motion estimation and motion compensation operations are applied to the video portion and the film portion. The film portion can be a background image, while the video portion can be a scrolling text image overlaid on the background image. In a series of input frames, because the film portion follows a regular input pattern while the video portion moves at a constant speed, traditional motion estimation and motion compensation operations often fail to simultaneously eliminate jitter in both the film and video portions, resulting in a smooth image. In contrast, the present invention processes both the film and video portions simultaneously to produce a jitter-free image.

[0099] First, the video processor determines whether each image block in the input frame is a mixed block containing image content from a video portion. If so, the temporal motion vector candidate for the image block is multiplied by a gain determined based on the positional difference of the video portion. The motion estimation unit then selects the optimal motion vector for the image block from a plurality of motion vector candidates, including the temporal motion vector candidate modified by the gain. The motion estimation unit then transmits the selected optimal motion vector to the motion compensation unit. The motion compensation unit then interpolates the interpolated frame using mixed phase coefficients determined based on the positional difference of the video portion to appropriately smooth the video portion and eliminate jitter. Conversely, if the image block is determined to be a non-mixed block, the motion estimation unit does not modify the temporal motion vector candidate and instead performs interpolation using normal phase coefficients generated based on the input frame pattern and the input / output frame rate.

[0100] As a result, during the motion estimation and motion compensation processes, the video processor of the present invention can accurately predict the block motions of the film portion and the video portion, thereby eliminating unnecessary jitter in the mixed-mode video and improving image quality.

[0101] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A motion estimation method for a video processor, characterized in that: The motion estimation method includes: receiving a current input frame, the current input frame including a film portion and a video portion; Calculating a frame difference between the current input frame and a first previous input frame that is located before the current input frame and adjacent to the current input frame; comparing the frame difference with a first threshold to generate a comparison result; When the comparison result indicates that the frame difference is less than the first threshold, determining whether a first block among the plurality of blocks in the current input frame is a mixed block according to a block difference value associated with the first block, wherein the block difference value indicates a difference between the first block in the current input frame and a block located at the same position as the first block in the first previous input frame; generating at least one temporal motion vector candidate for the first block; When the first block is determined to be the mixed block, modifying the at least one temporal motion vector candidate for the first block according to a current position difference and a previous position difference of the video portion of the current input frame; and determining a motion vector for the first block from a plurality of motion vector candidates, the plurality of motion vector candidates including the at least one temporal motion vector candidate; The current position difference is the difference between the position of the video portion of the current input frame corresponding to a current output frame and the position of a video portion of a second previous input frame, and the previous position difference is the difference between the position of the video portion of the current input frame corresponding to a previous output frame before the current output frame and the position of the video portion of the second previous input frame, wherein the second previous input frame is the last input frame in the previous frame group of the current input frame.

2. The motion estimation method according to claim 1, wherein: Also includes: An interpolated frame is generated based on the current input frame and the second previous input frame using a phase coefficient related to the current input frame, wherein a block on the interpolated frame is generated based on the first block on the current input frame and a second block on the second previous input frame indicated by the motion vector for the first block of the current input frame using the phase coefficient.

3. The motion estimation method according to claim 2, wherein: In the case where the first block is determined to be a mixed block, the phase coefficient is generated based on a ratio of a first position difference value to the current position difference value, wherein the first position difference value is the difference between the position of a video portion of the interpolated frame and the position of the video portion of the second previous input frame, and the current position difference value is the difference between the position of the video portion of the current input frame and the position of the video portion of the second previous input frame.

4. The motion estimation method according to claim 1, wherein: The step of modifying the at least one temporal motion vector candidate for the first block according to the current position difference and the previous position difference of the video portion of the current input frame comprises: Each of the at least one temporal motion vector is multiplied by a gain determined according to the current position difference and the previous position difference of the video portion of the second previous input frame to modify the at least one temporal motion vector candidate for the first block.

5. The motion estimation method according to claim 4, wherein: The gain is equal to a ratio of the current position difference to the previous position difference.

6. The motion estimation method according to claim 1, wherein: When the first block is determined not to be the mixed block, the at least one temporal motion vector candidate is not modified.

7. The motion estimation method according to claim 1, wherein: The step of determining whether the first block is the mixed block according to the block difference value of the first block among the plurality of blocks on the current input frame includes: summing up the difference between each pixel data in the first block on the current input frame and each pixel data in the block located at the same position as the first block on the first previous input frame to calculate the block difference value for the first block; determining a block mixing degree according to the block difference value with respect to the first block; and The block mixing degree is compared with a second threshold to determine whether the first block is the mixed block.

8. A video processor, characterized in that: Used to perform the following steps: receiving a current input frame, the current input frame including a film portion and a video portion; Calculating a frame difference between the current input frame and a first previous input frame that is located before the current input frame and adjacent to the current input frame; comparing the frame difference with a first threshold to generate a comparison result; When the comparison result indicates that the frame difference is less than the first threshold, determining whether a first block among the plurality of blocks in the current input frame is a mixed block according to a block difference value associated with the first block, wherein the block difference value indicates a difference between the first block in the current input frame and a block located at the same position as the first block in the first previous input frame; generating at least one temporal motion vector candidate for the first block; When the first block is determined to be the mixed block, modifying the at least one temporal motion vector candidate for the first block according to a current position difference and a previous position difference of the video portion of the current input frame; and determining a motion vector for the first block from a plurality of motion vector candidates, the plurality of motion vector candidates including the at least one temporal motion vector candidate; The current position difference is the difference between the position of the video portion of the current input frame corresponding to a current output frame and the position of a video portion of a second previous input frame, and the previous position difference is the difference between the position of the video portion of the current input frame corresponding to a previous output frame before the current output frame and the position of the video portion of the second previous input frame, wherein the second previous input frame is the last input frame in the previous frame group of the current input frame.

9. The video processor according to claim 8, wherein: The video processor also generates an interpolated frame based on the current input frame and the second previous input frame using a phase coefficient related to the current input frame, wherein a block on the interpolated frame is generated using the phase coefficient based on the first block on the current input frame and a second block on the second previous input frame indicated by the motion vector for the first block of the current input frame.

10. The video processor according to claim 9, wherein: In the case where the first block is determined to be a mixed block, the phase coefficient is generated based on a ratio of a first position difference value to the current position difference value, wherein the first position difference value is the difference between the position of a video portion of the interpolated frame and the position of the video portion of the second previous input frame, and the current position difference value is the difference between the position of the video portion of the current input frame and the position of the video portion of the second previous input frame.

11. The video processor according to claim 8, wherein: The step of modifying the at least one temporal motion vector candidate for the first block according to the current position difference and the previous position difference of the video portion of the current input frame comprises: Each of the at least one temporal motion vector is multiplied by a gain determined according to the current position difference and the previous position difference of the video portion of the second previous input frame to modify the at least one temporal motion vector candidate for the first block.

12. The video processor according to claim 11, wherein: The gain is equal to a ratio of the current position difference to the previous position difference.

13. The video processor according to claim 8, wherein When the first block is determined not to be the mixed block, the at least one temporal motion vector candidate is not modified.

14. The video processor according to claim 8, wherein: The step of determining whether the first block is the mixed block according to the block difference value of the first block among the plurality of blocks on the current input frame includes: summing up the difference between each pixel data in the first block on the current input frame and each pixel data in the block located at the same position as the first block on the first previous input frame to calculate the block difference value for the first block; determining a block mixing degree according to the block difference value with respect to the first block; and The block mixing degree is compared with a second threshold to determine whether the first block is the mixed block.

15. A motion compensation method for a video processor, characterized in that: The motion compensation method includes: receiving a current input frame, the current input frame including a film portion and a video portion; determining, based on a block difference value associated with a first block among a plurality of blocks in the current input frame, whether the first block is a mixed block, wherein the block difference value indicates a difference between the first block in the current input frame and a block located at the same position as the first block in a first previous input frame, the first previous input frame being located before and adjacent to the current input frame; When the first block is determined to be the mixed block, generating a phase coefficient for the first block according to a ratio of a first position difference to a second position difference, wherein the first position difference is a difference between a position of a video portion of an interpolated frame and a position of a video portion of a second previous input frame, and the second position difference is a difference between a position of the video portion of the current input frame and a position of the video portion of the second previous input frame, the second previous input frame being the last input frame in a frame group preceding the current input frame; and The interpolated frame is generated using the phase coefficient based on the current input frame and the second previous input frame, wherein a block on the interpolated frame is generated using the phase coefficient based on the first block on the current input frame and a second block on the second previous input frame indicated by a motion vector for the first block of the current input frame.

16. The motion compensation method according to claim 15, wherein: The motion vector is selected from a plurality of motion vector candidates, the plurality of motion vector candidates including at least one temporal motion vector candidate, and the at least one temporal motion vector candidate is modified according to the second position difference.

17. The motion compensation method according to claim 15, wherein: Also includes: Calculating a frame difference between the current input frame and the first previous input frame; comparing the frame difference with a first threshold to generate a comparison result; as well as When the comparison result indicates that the frame difference is less than the first threshold, it is determined whether the first block is the mixed block.

18. The motion compensation method according to claim 15, wherein: Also includes: Calculating a video step by dividing a difference between a position of a video portion of a last frame in a current frame group in which the current input frame is located and a position of a video portion of a frame in a previous frame group before the current frame group by a number of interpolated frames determined according to an output frame rate and an input frame rate of the video processor; as well as The position of the video portion of a previous frame is added to the video step to obtain the position of the video portion of the interpolated frame.

19. The motion compensation method according to claim 15, wherein: The step of determining whether the first block is the mixed block according to the block difference value of the first block among the plurality of blocks on the current input frame includes: summing up the difference between each pixel data in the first block on the current input frame and each pixel data in the block located at the same position as the first block on the first previous input frame to calculate the block difference value for the first block; determining a block mixing degree according to the block difference value with respect to the first block; and The block mixing degree is compared with a second threshold to determine whether the first block is the mixed block.

20. A video processor, characterized in that: Used to perform the following steps: receiving a current input frame, the current input frame including a film portion and a video portion; determining, based on a block difference value associated with a first block among a plurality of blocks in the current input frame, whether the first block is a mixed block, wherein the block difference value indicates a difference between the first block in the current input frame and a block located at the same position as the first block in a first previous input frame, the first previous input frame being located before and adjacent to the current input frame; When the first block is determined to be the mixed block, generating a phase coefficient for the first block according to a ratio of a first position difference to a second position difference, wherein the first position difference is a difference between a position of a video portion of an interpolated frame and a position of a video portion of a second previous input frame, and the second position difference is a difference between a position of the video portion of the current input frame and a position of the video portion of the second previous input frame, the second previous input frame being the last input frame in a frame group preceding the current input frame; and The interpolated frame is generated using the phase coefficient based on the current input frame and the second previous input frame, wherein a block on the interpolated frame is generated using the phase coefficient based on the first block on the current input frame and a second block on the second previous input frame indicated by a motion vector for the first block of the current input frame.

21. The video processor of claim 20, wherein: The motion vector is selected from a plurality of motion vector candidates, the plurality of motion vector candidates including at least one temporal motion vector candidate, and the at least one temporal motion vector candidate is modified according to the second position difference.

22. The video processor of claim 20, wherein: The video processor is also used to perform the following steps: Calculating a frame difference between the current input frame and the first previous input frame; comparing the frame difference with a first threshold to generate a comparison result; as well as When the comparison result indicates that the frame difference is less than the first threshold, it is determined whether the first block is the mixed block.

23. The video processor of claim 20, wherein: The video processor is also used to perform the following steps: Calculating a video step by dividing a difference between a position of a video portion of a last frame in a current frame group in which the current input frame is located and a position of a video portion of a frame in a previous frame group before the current frame group by a number of interpolated frames determined according to an output frame rate and an input frame rate of the video processor; as well as The position of the video portion of a previous frame is added to the video step to obtain the position of the video portion of the interpolated frame.

24. The video processor of claim 20, wherein: The step of determining whether the first block is the mixed block according to the block difference value of the first block among the plurality of blocks on the current input frame includes: summing up the difference between each pixel data in the first block on the current input frame and each pixel data in the block located at the same position as the first block on the first previous input frame to calculate the block difference value for the first block; determining a block mixing degree according to the block difference value with respect to the first block; and The block mixing degree is compared with a second threshold to determine whether the first block is the mixed block.

Images