Image processing method based on segmented graph and video processor thereof

Abstract

The invention discloses an image processing method for a video processor and a video processor thereof, the image processing method is used for generating an interpolation frame, and the method comprises the following steps: receiving a segment graph which comprises a plurality of segment marks; generating an original motion vector for interpolating a first block on the frame; determining whether to mark the first block as a segmentation boundary or not according to a plurality of segmentation marks in a region corresponding to the first block; generating a modified motion vector for the first block when the first block is marked as the segmentation boundary; judging whether the first block is located in a coated area or a non-coated area according to the corrected motion vector; and selecting a final motion vector from the original motion vector and the corrected motion vector for a specific pixel of the first block according to the segmentation map and according to the fact that the first block is judged to be located in the covered area or the non-covered area.

Description

Technical Field

[0001] The present invention relates to an image processing method for a video processor, and in particular to an image processing method based on a segmentation map and a related video processor. Background Art

[0002] Motion Estimation and Motion Compensation (MEMC) is a technology used for frame interpolation, which allows a series of image frames to be displayed at a higher frame rate. For example, if a 30 Hz original video (such as a movie) needs to be displayed at a frequency of 60 Hz, an interpolated frame must be added between every two adjacent input frames of the original video to achieve twice the frame rate. The image of the interpolated frame can be predicted using the motion vector (MV) between the current input frame and the previous input frame, so that the output video can be displayed smoothly.

[0003] Since the interpolated frame is composed of the original non-existent image generated based on the motion vector, the correctness of the motion vector is very important in the motion estimation and motion compensation operations. In the interpolated frame, if part of the interpolated image gets the wrong motion vector, it may cause the generation of halos, and halos are prone to appear at the boundaries of moving objects. Therefore, the industry is working to improve the accuracy of motion vector prediction, especially for blocks located at the boundaries of foreground objects. Summary of the invention

[0004] Therefore, the main purpose of the present invention is to propose a new image processing method that can be used in a video processor, so as to improve the accuracy of motion vector (MV) prediction.

[0005] An embodiment of the present invention discloses an image processing method for a video processor, which is used to generate an interpolated frame having a plurality of blocks according to a previous frame and a current frame. The method comprises the following steps: receiving a segmentation map, the segmentation map comprising a plurality of segmentation flags; generating an original motion vector (MV) for a first block among the plurality of blocks; determining whether to mark the first block as a segmentation boundary according to the plurality of segmentation flags in a region corresponding to the first block; generating a modified motion vector for the first block when the first block is marked as the segmentation boundary; determining whether the first block is located in a cover region or an uncover region according to the modified motion vector; and selecting a final motion vector from the original motion vector and the modified motion vector according to the segmentation map and according to the first block being determined to be located in the cover region or the uncover region, for a specific pixel of the first block.

[0006] Another embodiment of the present invention discloses a video processor for receiving a segmentation map including a plurality of segmentation flags to generate an interpolated frame having a plurality of blocks according to a previous frame and a current frame. The video processor includes a motion estimation (ME) circuit, a vector processing circuit, and a motion compensation (MC) circuit. The motion estimation circuit is used to generate an original motion vector for a first block among the plurality of blocks, and determine whether to mark the first block as a segmentation boundary according to the plurality of segmentation flags in a region corresponding to the first block. The vector processing circuit is coupled to the motion estimation circuit, and is used to generate a modified motion vector for the first block when the motion estimation circuit marks the first block as the segmentation boundary, and determine whether the first block is located in a covered area or a non-covered area according to the modified motion vector. The motion compensation circuit is coupled to the vector processing circuit and is used to select a final motion vector from the original motion vector and the modified motion vector for a specific pixel of the first block according to the segmentation map and according to whether the first block is determined by the vector processing circuit to be located in the covered area or the non-covered area.

[0007] Another embodiment of the present invention discloses a video processor for receiving a segmentation map including a plurality of segmentation flags to generate an interpolated frame having a plurality of blocks according to a previous frame and a current frame. The video processor includes a motion estimation circuit, a vector processing circuit, and a motion compensation circuit. The motion estimation circuit is used to generate an original motion vector for a first block among the plurality of blocks, and determine whether to mark the first block as a segmentation boundary according to the plurality of segmentation flags in a region corresponding to the first block. The vector processing circuit is coupled to the motion estimation circuit, and is used to generate a modified motion vector for the first block when the motion estimation circuit marks the first block as the segmentation boundary. The motion compensation circuit is coupled to the vector processing circuit, and is used to select a final motion vector from the original motion vector and the modified motion vector according to the segmentation map for a specific pixel of the first block. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 FIG. 1 is a schematic diagram of a display system according to an embodiment of the present invention.

[0009] Figure 2 A detailed implementation of a video processor is shown.

[0010] Figure 3 The figure is a flowchart of an image processing process according to an embodiment of the present invention.

[0011] Figure 4 A segment boundary marking operation for a block on an interpolated frame is shown.

[0012] Figure 5 The operation of obtaining a modified motion vector from a plurality of neighboring blocks is shown.

[0013] Figure 6 An exemplary embodiment of determining a covered area and a non-covered area is shown.

[0014] Figure 7 The motion compensation operation performed in the block marked as a segment boundary and determined to be located in the occluded area is shown.

[0015] The reference numerals are described as follows:

[0016] 10 Display System

[0017] 12 Video provider

[0018] 104 Frame Rate Converter

[0019] 106 Video Processor

[0020] 108 display driver

[0021] 110 Display

[0022] 202 Motion Estimation Circuit

[0023] 204 Vector Processing Circuit

[0024] 206 Motion Compensation Circuit

[0025] IMG_P Previous frame

[0026] IMG_C Current frame

[0027] MAP_P, MAP_C segment diagram

[0028] 30 Image Processing Process

[0029] Steps 302-312

[0030] IMG_I interpolation frame

[0031] Blocks B1, B2, C1, B3, B4, B5, B6

[0032] MV1, MV2 Best Bidirectional Motion Vector

[0033] A1_P, A1_C, A2_P, A2_C area

[0034] P3_P, P3_C, P4_P, P4_C, P51, P52, P61, P62 pixels

[0036] BV_org Raw motion vector

[0037] BV_new Corrected motion vector DETAILED DESCRIPTION

[0038] Figure 1 FIG. 1 is a schematic diagram of a display system 10 according to an embodiment of the present invention. Figure 1As shown, the display system 10 may be, for example, a television, which may receive and display an original video having a series of image frames provided by a video providing unit 12. The video providing unit 12 may be, for example, a digital video disc (DVD) player or a video streaming service provider, which may communicate with the display system 10 via a wired network or a wireless network. The display system 10 includes a video processor 106, a display driver 108, and a display screen 110. Generally speaking, the frame rate of the original video is different from the frame rate to be displayed on the display screen 110. The video processor 106 may include a television control integrated circuit and / or a frame rate converter integrated circuit (FRC IC) for converting the original video, which generally has a lower frame rate, to generate a series of output frames with a higher frame rate for display through the display screen 110. Alternatively, the video processor 106 may be a television control integrated circuit with a built-in frame rate conversion function, that is, Figure 1 The frame rate converter 104 shown can be considered as an independent frame rate conversion integrated circuit or a frame rate conversion circuit located in a TV control integrated circuit. For example, when the frame rate converter 104 receives a 24 Hz input video with two input frames A and B, the two input frames A and B can be converted into five image frames A, A, A, B, B at 60 Hz, and motion estimation and motion compensation (MEMC) can be performed based on the input frames to generate appropriate interpolated frames or extrapolated frames.

[0039] In another embodiment, the video processor 106 may include a graphics processing unit (GPU) and an independent frame rate conversion integrated circuit, or the video processor 106 may be a graphics processing unit including a built-in frame rate conversion circuit.

[0040] The display driver 108 may convert the image data into a data voltage signal, and drive the display screen 110 to display the image frame through the data voltage signal. The display driver 108 may include a timing controller, a source driver, a gate driver, and / or any other device that can be used to drive the display screen 110. The display screen 110 may be of any type, such as a liquid crystal display (LCD), a light-emitting diode (LED) display, or a plasma display panel (PDP), but is not limited thereto.

[0041] The algorithms for motion estimation and motion compensation can be developed based on a segmentation map, which is used to indicate whether each pixel on an image frame has a foreground image or a background image. For example, the segmentation map may include multiple segmentation flags, each of which has 1 bit of data and corresponds to a pixel. In other words, the segmentation map includes pixel-level data, each of which indicates the image information of the corresponding pixel, and the segmentation flags corresponding to a frame of pixels can form a segmentation map. The segmentation map can accurately and in detail define the foreground area and / or background area on an image frame at the pixel level. In an exemplary embodiment, in the segmentation map, the segmentation flag corresponding to the pixel in the background area is 0, while the segmentation flag corresponding to the pixel in the foreground area is 1.

[0042] The present invention proposes a novel image processing method, which can perform motion estimation and motion compensation based on the information provided by the segmentation map. The proposed image processing method can be implemented in a frame rate converter and / or a video processor, such as Figure 1 The frame rate converter 104 and / or the video processor 106 in the embodiment of the present invention are used to obtain the dehalo function. Based on the information of the segment map, the image processing method can provide a dehalo function (i.e., remove the halo on the interpolated image). Since the segment map has pixel-level information, using the segment map in motion estimation and motion compensation operations can perform dehalo more efficiently and improve the accuracy of motion vector (MV) prediction.

[0043] Figure 21 shows a detailed implementation of the video processor 106. The video processor 106 includes a motion estimation (ME) circuit 202, a vector processing circuit 204, and a motion compensation (MC) circuit 206. The video processor 106 can generate at least one interpolated frame according to a previous frame and a current frame, wherein each block on the interpolated frame can be predicted by selecting an optimal motion vector. Each block includes a plurality of pixels, and these pixels can be divided and allocated to different blocks in any appropriate manner. In general, the motion estimation circuit 202 can generate one or more optimal motion vectors (including a bidirectional motion vector, a forward motion vector, and / or a backward motion vector) for each block. For each block, the motion estimation circuit 202 can calculate the Sum of Absolute Difference (SAD) of the image in the area projected by each motion vector on the previous frame and the current frame to find the motion vector with the smallest SAD, thereby generating the optimal motion vector for this block. Alternatively or additionally, a cost function including a sum of absolute errors may be used to determine the similarity of the images projected to each motion vector, so as to select the best motion vector from a plurality of candidate motion vectors. The vector processing circuit 204 is coupled to the motion estimation circuit 202, and may be used to process the received bidirectional motion vectors, the forward motion vectors, and the backward motion vectors to generate one or more final motion vectors, and output the final motion vectors to the motion compensation circuit 206. Then, the motion compensation circuit 206 generates an image in each block according to the corresponding final motion vectors, thereby generating an interpolated frame.

[0044] Figure 3 FIG. 3 is a flow chart of an image processing process 30 according to an embodiment of the present invention. The image processing process 30 may be implemented in a video processor, such as Figure 1 and Figure 2 The video processor 106 in FIG. 1 is used to generate an interpolated frame according to a previous frame and a current frame. Figure 3 As shown, the image processing flow 30 includes the following steps:

[0045] Step 302: Receive a segment map, which includes a plurality of segment marks.

[0046] Step 304: Generate an original motion vector for a first block in the interpolated frame.

[0047] Step 306: Determine whether to mark the first block as a segmentation boundary according to a plurality of segmentation flags in a region corresponding to the first block.

[0048] Step 308: Generate a modified motion vector for the first block when the first block is marked as a segment boundary.

[0049] Step 310: Determine whether the first block is located in a covered area or an uncovered area according to the modified motion vector.

[0050] Step 312: According to the segmentation map and according to whether the first block is determined to be located in the coverage area or the non-coverage area, select a final motion vector from the original motion vector and the modified motion vector for a specific pixel of the first block.

[0051] According to the image processing flow 30, the motion estimation circuit 202, the vector processing circuit 204 and the motion compensation circuit 206 in the video processor 106 can receive the segmentation map and perform respective operations according to the segmentation map. It should be noted that the steps of the image processing flow 30 can be applied to all blocks or part of the blocks on the interpolation frame in the same way, wherein the first block is used as an example for description.

[0052] Please refer to Figure 3 Matching Figure 2 As shown, since the video processor 106 uses a previous frame IMG_P and a current frame IMG_C to perform image interpolation, it can receive a segmentation map MAP_P of the previous frame IMG_P and a segmentation map MAP_C of the current frame IMG_C.

[0053] First, the motion estimation circuit 202 may use the segment maps MAP_P and MAP_C to perform “segment boundary marking.” As described above, the motion estimation circuit 202 may generate several best motion vectors for each block, including a forward motion vector, a backward motion vector and / or a bidirectional motion vector.

[0054] For example, the best bidirectional motion vector of the first block may be selected from a plurality of candidate motion vectors. The motion estimation circuit 202 may project each candidate motion vector to the previous frame IMG_P and / or the current frame IMG_C to obtain an image in a first region on the previous frame IMG_P to which the candidate motion vector is projected and an image in a second region on the current frame IMG_C to which the candidate motion vector is projected, and calculate the sum of absolute errors between the image in the first region and the image in the second region. The motion estimation circuit 202 may obtain the sum of absolute errors corresponding to each candidate motion vector, and obtain the bidirectional motion vector with the minimum sum of absolute errors as the best bidirectional motion vector. Then, the motion estimation circuit 202 outputs the best bidirectional motion vector to the vector processing circuit 204.

[0055] As described above, the motion estimation circuit 202 may also obtain the segmentation map MAP_P of the previous frame IMG_P and the segmentation map MAP_C of the current frame IMG_C, and each of the segmentation maps MAP_P and MAP_C has a plurality of segmentation flags for indicating pixel states. For the best bidirectional motion vector, when the image contents of the first region and the second region are used to calculate the absolute error sum, the motion estimation circuit 202 may also obtain the segmentation flag in the first region and the segmentation flag in the second region, and use the segmentation flags in these regions to perform segmentation boundary marking, that is, to determine whether to mark the first block as a segmentation boundary.

[0056] In one embodiment, when calculating the Sum of Absolute Errors, the motion estimation circuit 202 may obtain the segmentation flag in the region to which each candidate motion vector is projected, and obtain the segmentation flag corresponding to the best bidirectional motion vector to perform segmentation boundary marking. Alternatively, the motion estimation circuit 202 may first determine the best bidirectional motion vector, and then use the projection result of the best bidirectional motion vector to determine the region for obtaining the segmentation flag for performing segmentation boundary marking. The regions for obtaining the segmentation flag may be the same as or different from the first region and the second region used to calculate the Sum of Absolute Errors and determine the best bidirectional motion vector.

[0057] Figure 4 FIG. 4 shows a segment boundary marking operation for blocks B1 and B2 on an interpolated frame IMG_I, which is generated based on a previous frame IMG_P and a current frame IMG_C. Figure 4 As shown, these image frames include a background image moving downwards, which is covered by a foreground object moving upwards, and the interpolated frame IMG_I is generated by predicting the motion of the moving object in motion estimation and motion compensation operations. In this example, the motion estimation circuit 202 can generate the best bidirectional motion vectors MV1 and MV2 for blocks B1 and B2 respectively.

[0058] like Figure 4 As shown, the best bidirectional motion vector MV1 of the block B1 is projected onto the previous frame IMG_P and the current frame IMG_C to obtain an area A1_P on the previous frame IMG_P and an area A1_C on the current frame IMG_C. Each area A1_P and A1_C includes M pixels. Among the total 2M pixels of the areas A1_P and A1_C, the motion estimation circuit 202 can refer to the segmentation maps MAP_P and MAP_C for the previous frame IMG_P and the current frame IMG_C, and determine how many segmentation flags in the areas A1_P and A1_C indicate the foreground, thereby calculating the number of pixels in the areas A1_P and A1_C marked as the foreground image.

[0059] Similarly, the best bidirectional motion vector MV2 of block B2 is projected onto the previous frame IMG_P to obtain an area A2_P, and onto the current frame IMG_C to obtain an area A2_C, wherein each area A2_P and A2_C also includes M pixels. By referring to the segmentation maps MAP_P and MAP_C, the motion estimation circuit 202 can determine how many segmentation flags in the areas A2_P and A2_C indicate the foreground, thereby calculating the number of pixels in the areas A2_P and A2_C marked as the foreground image.

[0060] If a block on the interpolated frame is completely located in the background area, the area projected by the best bidirectional motion vector of the block on the previous frame and the current frame is usually located in the background area, and therefore, the number of segmentation flags indicating foreground may be equal to or close to 0. Conversely, if a block on the interpolated frame is completely located in the foreground area, the area projected by the best bidirectional motion vector of the block on the previous frame and the current frame is usually located in the foreground area, and therefore, the number of segmentation flags indicating foreground may be equal to or close to 2M (i.e., the total number of pixels in these areas). In addition, if a block on the interpolated frame is located at or close to the boundary of a foreground image, the area projected by the best bidirectional motion vector of the block on the previous frame and the current frame usually includes both the foreground area and the background area, and therefore, the number of segmentation flags indicating foreground may have an intermediate value between 0 and 2M.

[0061] In one embodiment, the motion estimation circuit 202 may define a low threshold TH1 and a high threshold TH2 for the segment boundary marking. In the area projected by the best bidirectional motion vector of a block, the motion estimation circuit 202 may calculate the number of pixels marked as foreground images according to the segment map. If the number of pixels marked as foreground images is between the low threshold TH1 and the high threshold TH2, it may be determined that the block is marked as a segment boundary, which indicates the boundary of a foreground object. In addition, if the number of pixels marked as foreground images is less than the low threshold TH1, the block may be determined to be a background block; and if the number of pixels marked as foreground images is greater than the high threshold TH2, the block may be determined to be a foreground block. The low threshold TH1 and the high threshold TH2 may be set to any value between 0 and the total number of pixels in the relevant area (e.g., 2M in the above embodiment), wherein the high threshold TH2 is greater than the low threshold TH1.

[0062] Please refer to Figure 4For block B1, its corresponding area A1_P on the previous frame IMG_P is the background area, and its corresponding area A1_C on the current frame IMG_C is the foreground area. Therefore, the total number of pixels marked as foreground by the segmentation flag in areas A1_P and A1_C is approximately equal to M, which is an intermediate value between 0 and 2M; therefore, block B1 can be determined to be located at the boundary of a foreground object and marked as a segmentation boundary. For block B2, its corresponding area A2_P on the previous frame IMG_P includes both background and foreground images, and its corresponding area A2_C on the current frame IMG_C also includes both background and foreground images. Therefore, the total number of pixels marked as foreground by the segmentation flag in areas A2_P and A2_C may be an intermediate value between 0 and 2M; therefore, block B2 can be determined to be located at the boundary of a foreground object and marked as a segmentation boundary. In the existing interpolated image, the boundaries of the moving foreground objects are the locations where halo is likely to appear. Therefore, the motion estimation circuit 202 can mark the blocks at these locations to facilitate the subsequent de-halo operation.

[0063] Then, the motion estimation circuit 202 may transmit information of the block marked as a segment boundary to the vector processing circuit 204, such as Figure 2 The vector processing circuit 204 may perform a "segmentation occlusion process" based on the information provided by the motion estimation circuit 202. The operation of the "segmentation occlusion process" includes two main steps: first, the vector processing circuit 204 may perform motion vector correction in the occlusion area in the segment boundary, which is not easy to be processed by the existing motion estimation and motion compensation algorithms; second, for the blocks in the occlusion area, the vector processing circuit 204 may determine whether each block is located in a covered area or a non-covered area.

[0064] For the first block marked as a segment boundary, if suitable motion vectors can be taken from multiple neighboring blocks to modify the first block, the first block can be considered to be located in an occluded area. Figure 5 According to the movement of a foreground object from the previous frame IMG_P to the current frame IMG_C, the foreground object can be generated on the interpolated frame IMG_I through a foreground motion vector. Figure 5 As shown, each grid on the interpolated frame IMG_I can be regarded as a block, wherein the block with a high-density dotted grid background represents the foreground area, the blank block represents the background area, and the block with a low-density dotted grid background is located at the boundary of the foreground object and is therefore marked as a segmentation boundary.

[0065] As described above, the motion estimation circuit 202 can obtain the best bidirectional motion vector for each block on the interpolated frame IMG_I, and transmit the best bidirectional motion vector information of each block to the vector processing circuit 204. Therefore, for the first block marked as a segment boundary and determined to be located in the occlusion area, the vector processing circuit 204 can generate a modified motion vector for the first block.

[0066] like Figure 5 As shown, for a block C1 marked as a segment boundary, multiple neighboring blocks located around the block C1 will be taken into consideration. In this example, the vector processing circuit 204 may mark an area centered on the block C1 and including N×N blocks, where N may be any positive integer. Then, the vector processing circuit 204 may obtain the best bidirectional motion vector of each block in the N×N neighboring blocks as a candidate motion vector to generate a candidate pool. The vector processing circuit 204 then selects a modified motion vector for the block C1 from the candidate motion vectors included in the candidate pool.

[0067] The modified motion vector should meet certain criteria. Since the purpose of the dehalation operation is to remove unnecessary halos around the borders of moving foreground objects, where halos may be caused by taking an erroneous foreground motion vector in the background area near the foreground object, and the dehalation operation replaces the erroneous foreground motion vector with a correct background motion vector. In this case, the modified motion vector determined by the vector processing circuit 204 should be a background motion vector for the block in the background image. For example, in Figure 5 In FIG. 5 , the block located at the upper right among the N×N adjacent blocks can be used to generate a background motion vector to be selected as the modified motion vector for the block C1.

[0068] Therefore, it is necessary to determine whether each block in the N×N adjacent blocks is a background block, so as to obtain a suitable modified motion vector. In one embodiment, this operation can be completed by the motion estimation circuit 202 according to the segmentation map, wherein the motion estimation circuit 202 calculates the number of pixels in the projection area marked as foreground by the segmentation flag, and compares the calculated number with a low threshold and a high threshold to determine whether each block belongs to the foreground, background, or segmentation boundary. As described above, if the number of pixels marked as foreground is less than the low threshold, the block can be determined as a background block. In this case, the modified motion vector should be selected from the candidate motion vectors for the adjacent blocks determined to be background blocks (according to the segmentation map).

[0069] However, there may be multiple background blocks around the block C1, so there are multiple candidate motion vectors that can be selected as the modified motion vector. In this case, a cost function can be used to select and determine the best modified motion vector.

[0070] In one embodiment, the cost function is related to the distance between block C1 and each adjacent block. For example, the distance between block C1 and an adjacent block can be considered as a factor used to calculate the cost function. If a motion vector is used for an adjacent background block closer to block C1, the value of the cost function of the motion vector may be smaller; if a motion vector is used for an adjacent background block farther from block C1, the value of the cost function of the motion vector may be larger. Generally speaking, a motion vector used for an adjacent block closer to block C1 may be more reliable and thus have a greater chance of being selected as a modified motion vector.

[0071] Alternatively or additionally, the cost function is related to the absolute error sum of the candidate motion vectors as options for the modified motion vector. As described above, the motion estimation circuit 202 can calculate the absolute error sum of each bidirectional motion vector to determine the best bidirectional motion vector for each block. Therefore, each motion vector can be assigned a value of the absolute error sum, which indicates the similarity between the previous image and the current image to which the motion vector is projected, and is used to indicate the correctness of the motion vector. Therefore, in addition to outputting the best bidirectional motion vector, the motion estimation circuit 202 can also transmit the absolute error sum of each best bidirectional motion vector to the vector processing circuit 204, so that the vector processing circuit 204 can select the modified motion vector according to the absolute error sum value. In this case, the absolute error sum of each best bidirectional motion vector can be regarded as an element used to calculate the cost function. For example, if a motion vector has a smaller absolute error sum, the value of the cost function of this motion vector may be smaller; if a motion vector has a larger absolute error sum, the value of the cost function of this motion vector may be larger. Generally speaking, a motion vector with a smaller absolute error sum may be more reliable and thus has a greater chance of being selected as a modified motion vector.

[0072] In one embodiment, the cost function may be the sum of the distance parameter and the absolute error sum parameter, and the vector processing circuit 204 may select the background motion vector with the minimum cost function as the modified motion vector, which may have a shorter distance and / or a smaller absolute error sum with respect to the block C1. In this case, the distance of the block and the absolute error sum value of the motion vector may be taken into consideration at the same time to determine the modified motion vector, thereby achieving a more comprehensive determination result.

[0073] In addition to obtaining the corrected motion vector for each block in the masked area, the vector processing circuit 204 can also determine whether each block is located in a covered area or a non-covered area. "Covered" and "non-covered" represent situations where foreground and background objects are interlaced. A covered area is an area where the background image was originally visible in the previous frame but is covered by the foreground object, and a non-covered area is an area where the background image was originally covered by the foreground object and appears in the subsequently generated interpolated frame. Covered and non-covered areas have interlaced foreground objects and background images, so halos are likely to appear in these areas. Motion compensation can be performed in different ways in the covered area and the non-covered area.

[0074] In one embodiment, the vector processing circuit 204 may use the segmentation map to determine whether the block in the occlusion area is in the coverage area or the non-coverage area according to the modified motion vector. Figure 5 As shown, after obtaining the modified motion vector for block C1 , the vector processing circuit 204 may project the modified motion vector to the previous frame IMG_P and the current frame IMG_C to obtain the position of the modified motion vector projection.

[0075] Figure 6 An exemplary embodiment of determining the covered area and the non-covered area is shown. Figure 6 As shown, the image on the interpolation frame IMG_I can be predicted and generated based on the previous frame IMG_P and the current frame IMG_C. These image frames include a background image moving downward, which is covered by a foreground object moving upward. The interpolation frame IMG_I includes two blocks B3 and B4, which are located in the occlusion area defined by the vector processing circuit 204. Figure 6 It can be seen that block B3 is located in the covered area, where the upward moving object covers the original background area; block B4 is located in the non-covered area, which was originally covered by the foreground object in the previous frame IMG_P, but appears in the interpolated frame IMG_I.

[0076] In order to determine the covered area and the non-covered area, the modified motion vector of the block B3 can be projected to the previous frame IMG_P and the current frame IMG_C. It should be noted that the modified motion vector is selected from a background motion vector, which has a downward motion. The vector processing circuit 204 can obtain a pixel P3_P located on the previous frame IMG_P and a pixel P3_C located on the current frame IMG_C projected by the modified motion vector. Referring to the corresponding segmentation maps MAP_P and MAP_C, the vector processing circuit 204 can obtain that the segmentation flag of the pixel P3_P indicates a background image and the segmentation flag of the pixel P3_C indicates a foreground image. In this case, the vector processing circuit 204 can determine that the block B3 is located in the covered area.

[0077] In a similar manner, the modified motion vector of block B4 can be projected to the previous frame IMG_P to obtain a pixel P4_P, and projected to the current frame IMG_C to obtain a pixel P4_C. Referring to the segmentation maps MAP_P and MAP_C, the vector processing circuit 204 can obtain that the segmentation flag of pixel P4_P indicates a foreground image and the segmentation flag of pixel P4_C indicates a background image. In this case, the vector processing circuit 204 can determine that block B4 is located in the non-encapsulated area.

[0078] In one embodiment, by referring to the segment maps MAP_P and MAP_C, the vector processing circuit 204 may determine that the pixels projected by the corrected motion vector on the previous frame IMG_P and the pixels projected by the corrected motion vector on the current frame IMG_C are both located in the background area or both located in the foreground area. In this case, the correction of the motion vector may be ignored to avoid unnecessary side effects.

[0079] It is worth noting that in the above-mentioned embodiment, the corrected motion vector is used for projection to find the segmentation mark of a pixel on the previous frame IMG_P and the segmentation mark of a pixel on the current frame IMG_C, so as to determine the covered area and the non-covered area. In another embodiment, the judgment of covered / non-covered can also be performed by projecting the corrected motion vector to an area with multiple pixels, while considering the segmentation marks of these pixels. For example, if the area on the previous frame IMG_P where the corrected motion vector is projected has more background pixels and the area on the current frame IMG_C where the corrected motion vector is projected has more foreground pixels, it can be determined that the block of the corrected motion vector is located in the covered area. In other words, Figure 6 P3_P, P3_C, P4_P and P4_C in can all be regarded as a region or even a block, in which the segmentation flags of multiple pixels can be used to perform coverage / non-coverage judgment.

[0080] Thus, the vector processing circuit 204 may output information of the covered and non-covered areas of each block in the occluded area to the motion compensation circuit 206. In addition, for each block in the occluded area, the vector processing circuit 204 may output a modified motion vector to the motion compensation circuit 206. It should be noted that in addition to the modified motion vector, the vector processing circuit 204 also outputs an original motion vector to the motion compensation circuit 206. In one embodiment, the best bidirectional motion vector determined by the motion estimation circuit 202 for each block may be regarded as the original motion vector, which may be provided to the vector processing circuit 204 and output to the motion compensation circuit 206. As described above, the modified motion vector is selected from the background motion vector, and the original motion vector is usually the foreground motion vector. The motion compensation circuit 206 is required to obtain the correct final motion vector from the modified motion vector and the original motion vector for the area on the boundary between the background image and the foreground object.

[0081] Due to the limitation of hardware cost, the existing motion estimation and motion compensation process processes image interpolation in units of one block. That is, the motion estimation and vector processing operations are used to generate the best motion vector for a block, and in the motion compensation operation, all pixels in the block use the same best motion vector to generate image content. The image generated in this way may be blocky, so a deblocking algorithm can be used to smooth the image to generate the final interpolated frame. For the block on the boundary of both the foreground and background images, at least some pixels have the wrong motion vector for interpolation. More specifically, if the final motion vector is a background motion vector, some pixels that should display the foreground will generate an erroneous image, thus causing the foreground boundary to be broken; or, if the final motion vector is a foreground motion vector, some pixels that should display the background will generate an erroneous image, causing the foreground boundary to be abnormally enlarged. After processing with the deblocking algorithm, the boundary area of ​​the image may be blurred, resulting in a halo phenomenon.

[0082] To solve this problem, in the present invention, the motion compensation circuit 206 performs compensation by referring to the segment map, and the segment map has segment flags based on each pixel, wherein each flag indicates the foreground or background state of a pixel. This pixel-level data enables the motion compensation circuit 206 to process the foreground pixels and background pixels in the same block in different ways to perform pixel-level compensation. In other words, the foreground pixels and background pixels in the same block can be interpolated using different motion vectors, thereby achieving a clear and correct interpolated image in the boundary area.

[0083] Figure 7 FIG. 2 shows motion compensation operations performed in blocks B5 and B6 marked as segment boundaries and determined to be located in the occluded area. Figure 7 In the example, the image on the interpolation frame IMG_I can be predicted and generated based on the previous frame IMG_P and the current frame IMG_C, and these image frames include a background image moving downward, which is covered by a foreground object moving upward. Blocks B5 and B6 on the interpolation frame IMG_I can be used as an example to illustrate the execution of the motion compensation operation. In this example, block B5 is located in the covered area and block B6 is located in the non-covered area, which can be determined by the vector processing circuit 204 based on the projection of the modified motion vector and the segment map, as described above.

[0084] Block B5 includes two pixels P51 and P52, wherein pixel P51 should preferably be predicted using a background motion vector, and pixel P52 should preferably be predicted using a foreground motion vector. The motion vector BV_org represents the original motion vector, which may be the best bidirectional motion vector determined by the motion estimation circuit 202. The motion vector BV_new represents the modified motion vector, which may be the best bidirectional motion vector selected from the neighboring blocks by the vector processing circuit 204. In the motion compensation operation, each pixel may obtain its image from the previous frame IMG_P or the current frame IMG_C according to the segmentation map.

[0085] In this example, the original motion vector BV_org may be used to determine how to predict the image of the pixel. For a specific pixel determined to be located in a block of the coverage area, the motion compensation circuit 206 may project the original motion vector BV_org to the current frame IMG_C to determine whether the pixel projected by the original motion vector BV_org on the current frame IMG_C has a foreground image or a background image, which may be indicated by the segmentation flag corresponding to the projected pixel in the segmentation map MAP_C of the current frame IMG_C. If the segmentation flag indicates that the projected pixel has a foreground image, the motion compensation circuit 206 may select the original motion vector BV_org as the final motion vector to generate the image of the specific pixel; if the segmentation flag indicates that the projected pixel has a background image, the motion compensation circuit 206 may select the modified motion vector BV_new as the final motion vector to generate the image of the specific pixel.

[0086] exist Figure 7 In the embodiment of , block B5 is determined to be located in the covered area. Pixel P51 in block B5 has an original motion vector BV_org, which is projected to a pixel on the current frame IMG_C, and this pixel is marked as a background pixel by the corresponding segmentation flag. Therefore, for pixel P51, it can be decided to select the modified motion vector BV_new as the final motion vector, wherein the modified motion vector BV_new is a background motion vector, which can be a bidirectional motion vector selected from the background block adjacent to block B5. Pixel P52 in block B5 has an original motion vector BV_org, which is projected to a pixel on the current frame IMG_C, and this pixel is marked as a foreground pixel by the corresponding segmentation flag. Therefore, for pixel P52, it can be decided to select the original motion vector BV_org as the final motion vector, wherein the original motion vector BV_org is a foreground motion vector, which can be the best bidirectional motion vector of block B5 determined by the motion estimation circuit 202.

[0087] Motion compensation for a block in a non-encapsulated area may be performed in different ways. For a specific pixel in a block determined to be located in a non-encapsulated area, the motion compensation circuit 206 may project the original motion vector BV_org to the previous frame IMG_P to determine whether the pixel projected by the original motion vector BV_org on the previous frame IMG_P has a foreground image or a background image, which may be indicated by a segmentation flag corresponding to the projected pixel in the segmentation map MAP_P of the previous frame IMG_P. If the segmentation flag indicates that the projected pixel has a foreground image, the motion compensation circuit 206 may select the original motion vector BV_org as the final motion vector to generate the image of the specific pixel; if the segmentation flag indicates that the projected pixel has a background image, the motion compensation circuit 206 may select the modified motion vector BV_new as the final motion vector to generate the image of the specific pixel.

[0088] exist Figure 7 In the embodiment of , block B6 is determined to be located in the non-encapsulated area. Pixel P61 in block B6 has an original motion vector BV_org, which is projected to a pixel on the previous frame IMG_P, and this pixel is marked as a foreground pixel by the corresponding segmentation flag. Therefore, for pixel P61, it can be decided to select the original motion vector BV_org as the final motion vector, wherein the original motion vector BV_org is a foreground motion vector, which can be the best bidirectional motion vector of block B6 determined by the motion estimation circuit 202. Pixel P62 in block B6 has an original motion vector BV_org, which is projected to a pixel on the previous frame IMG_P, and this pixel is marked as a background pixel by the corresponding segmentation flag. Therefore, for pixel P62, it can be decided to select the modified motion vector BV_new as the final motion vector, wherein the modified motion vector BV_new is a background motion vector, which can be a bidirectional motion vector selected from the background block adjacent to block B6.

[0089] Therefore, in a block located at the boundary of a foreground object on the interpolated frame, pixels determined to be background can use the modified motion vector to predict their image, while pixels determined to be foreground can use the original motion vector to predict their image. Depending on whether the block is located in the encapsulated area or the non-encapsulated area, the foreground and background pixels can be determined by projecting the original motion vector to the previous frame or the current frame and referring to the corresponding segment map. In this way, each pixel in the block can obtain the correct motion vector for interpolation, thereby generating a correct and clear image, while removing the halo in the occluded area.

[0090] It is worth noting that the purpose of the present invention is to propose an image processing method based on a segmented graph, which can be used to remove halo. Those skilled in the art can make modifications or changes accordingly, but are not limited to this. For example, in the present invention, motion estimation and motion compensation are performed based on the segmented graph, and the segmented graph can be an arbitrary and suitable information bitmap generated by a detection module using artificial intelligence (AI) or any other algorithm. The detection module can be a human detector, a face detector, or a detector that can be used to detect any possible foreground object. As long as the detection module can detect the content of the picture and generate pixel-level information according to the detection result, it can be applied to the image processing method of the present invention.

[0091] As described above, the existing motion estimation and motion compensation process processes image interpolation in units of a block, and generates an optimal motion vector for each block. Therefore, the block located at the boundary of the moving foreground object includes multiple different images, and thus appears blurred, and halos are likely to appear in this boundary area. In contrast, the motion estimation and motion compensation proposed by the present invention can perform motion compensation at the pixel level based on the segmentation map. The segmentation map includes multiple segmentation flags, each of which is used to indicate the foreground or background state of a pixel. Therefore, in a block on the interpolated frame, each foreground pixel can obtain the correct foreground motion vector and each background pixel can obtain the correct background motion vector to perform interpolation. Since all pixels can obtain their correct final motion vectors, halos will not appear on the interpolated frame.

[0092] In summary, the present invention proposes a novel image processing method and a related video processor for removing halo and improving the accuracy of motion vector prediction. Motion estimation and motion compensation can be performed based on a segmentation map, which includes pixel-level data for indicating foreground and background information of each pixel. The video processor may include a motion estimation circuit, a vector processing circuit, and a motion compensation circuit, wherein each circuit may receive the segmentation map and perform respective operations based on the segmentation map to generate an interpolated image.

[0093] The motion estimation circuit is used to generate an optimal bidirectional motion vector for each block, and determine whether each block should be marked as a segment boundary according to the segment flag corresponding to each area. The vector processing circuit is used to generate a modified motion vector for a block when a block is marked as a segment boundary, wherein the modified motion vector is selected from an optimal bidirectional motion vector of an adjacent block. The vector processing circuit also determines whether each block is located in a covered area or a non-covered area according to the modified motion vector. The obtained modified motion vector can be provided to the motion compensation circuit, and the optimal bidirectional motion vector determined by the motion estimation circuit can be regarded as an original motion vector to be provided to the motion compensation circuit. The motion compensation circuit selects a final motion vector for each pixel from the original motion vector and the modified motion vector according to the segment flag corresponding to each pixel in the block in the segment map, wherein the motion compensation circuit can obtain the corresponding segment flag according to whether the pixel in a block belongs to the covered area or the non-covered area by referring to the current frame or the previous frame. The motion estimation and motion compensation operations based on the segment map enable each pixel to perform image prediction using the correct motion vector, thereby removing the halo on the interpolated image.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image processing method for a video processor, for generating an interpolated frame having a plurality of blocks according to a previous frame and a current frame, characterized in that: The method includes: receiving a segment map, the segment map comprising a plurality of segment flags; generating an original motion vector for a first block among the plurality of blocks; Determining whether to mark the first block as a segment boundary according to the plurality of segment marks in an area corresponding to the first block; When the first block is marked as the segment boundary, generating a modified motion vector for the first block; According to the modified motion vector, determining whether the first block is located in a covered area or a non-covered area; and A final motion vector is selected from the original motion vector and the modified motion vector for a specific pixel of the first block according to the segmentation map and according to whether the first block is determined to be located in the coverage area or the non-coverage area.

2. The image processing method according to claim 1, characterized in that: Each of the plurality of segmentation flags indicates that a pixel on the previous frame or the current frame has a foreground image or a background image.

3. The image processing method according to claim 1, characterized in that: The area corresponding to the first block includes a first area of ​​the original motion vector projection used for the first block on the previous frame and a second area of ​​the original motion vector projection used for the first block on the current frame.

4. The image processing method according to claim 3, characterized in that: The step of determining whether to mark the first block as the segment boundary according to the plurality of segment marks in the area corresponding to the first block comprises: Calculate the number of pixels in the first region and the second region marked as a foreground image by the segmentation map; and When the number of pixels marked as the foreground image is between a first threshold and a second threshold, it is determined to mark the first block as the segment boundary.

5. The image processing method according to claim 4, characterized in that: Also includes: When the number of pixels marked as the foreground image in the first area and the second area is less than the first threshold, the first block is determined to be a background block.

6. The image processing method according to claim 1, characterized in that: The step of generating the modified motion vector for the first block comprises: The modified motion vector for the first block is selected from a plurality of candidate motion vectors for a plurality of neighboring blocks around the first block.

7. The image processing method according to claim 6, characterized in that: The modified motion vector is selected from one of the plurality of candidate motion vectors, and the selected candidate motion vector is used for an adjacent block among the plurality of adjacent blocks that is determined to be a background block according to the segmentation map.

8. The image processing method according to claim 6, characterized in that: The modified motion vector is selected based on a cost function related to at least one of: a distance between the first block and each of the plurality of neighboring blocks, and a sum of absolute errors of each of the plurality of candidate motion vectors.

9. The image processing method according to claim 1, characterized in that: According to the corrected motion vector, The step of determining whether the first block is located in the covering area or the non-covering area includes: When a first pixel projected by the modified motion vector on the current frame is indicated by a first segmentation mark among the multiple segmentation marks as having a foreground image, and a second pixel projected by the modified motion vector on the previous frame is indicated by a second segmentation mark among the multiple segmentation marks as having a background image, it is determined that the first block is located in the coverage area.

10. The image processing method according to claim 1, wherein: According to the corrected motion vector, The step of determining whether the first block is located in the covering area or the non-covering area includes: When a first pixel projected by the modified motion vector on the current frame is indicated by a first segmentation mark among the multiple segmentation marks as having a background image, and a second pixel projected by the modified motion vector on the previous frame is indicated by a second segmentation mark among the multiple segmentation marks as having a foreground image, it is determined that the first block is located in the non-encapsulated area.

11. The image processing method according to claim 1, wherein: The first block is determined to be located in the covered area, and the step of selecting the final motion vector from the original motion vector and the modified motion vector for the specific pixel of the first block includes: Determining, based on an indication of a corresponding third segmentation flag among the plurality of segmentation flags, whether a third pixel projected by the original motion vector on the current frame has a foreground image or a background image; and When the third segment flag indicates that the third pixel has the foreground image, the original motion vector is selected as the final motion vector, or when the third segment flag indicates that the third pixel has the background image, the modified motion vector is selected as the final motion vector.

12. The image processing method according to claim 1, characterized in that: The first block is determined to be located in the non-covered area, and the step of selecting the final motion vector from the original motion vector and the modified motion vector for the specific pixel of the first block includes: Determining, based on an indication of a corresponding fourth segmentation flag among the plurality of segmentation flags, whether a fourth pixel projected by the original motion vector on the previous frame has a foreground image or a background image; and When the fourth segment flag indicates that the fourth pixel has the foreground image, the original motion vector is selected as the final motion vector, or when the fourth segment flag indicates that the fourth pixel has the background image, the modified motion vector is selected as the final motion vector.

13. A video processor, characterized in that: The video processor is used to receive a segmentation map including a plurality of segmentation marks to generate an interpolated frame having a plurality of blocks according to a previous frame and a current frame, the video processor comprising: a motion estimation circuit for generating an original motion vector for a first block among the plurality of blocks, and according to the plurality of segment flags in a region corresponding to the first block, determining whether to mark the first block as a segment boundary; a vector processing circuit coupled to the motion estimation circuit, configured to generate a modified motion vector for the first block when the motion estimation circuit marks the first block as the segment boundary, and determine whether the first block is located in a covered area or a non-covered area according to the modified motion vector; and A motion compensation circuit is coupled to the vector processing circuit, and is used to select a final motion vector from the original motion vector and the modified motion vector for a specific pixel of the first block according to the segmentation map and according to whether the first block is determined by the vector processing circuit to be located in the covered area or the non-covered area.

14. The video processor of claim 13, wherein: Each of the plurality of segmentation flags indicates that a pixel on the previous frame or the current frame has a foreground image or a background image.

15. The video processor of claim 13, wherein: The area corresponding to the first block includes a first area of ​​the original motion vector projection used for the first block on the previous frame and a second area of ​​the original motion vector projection used for the first block on the current frame.

16. The video processor of claim 15, wherein: The motion estimation circuit calculates the number of pixels in the first area and the second area marked as a foreground image by the segmentation map, and determines to mark the first block as the segmentation boundary when the number of pixels marked as the foreground image is between a first threshold and a second threshold.

17. The video processor of claim 16, wherein: When the number of pixels marked as the foreground image in the first area and the second area is less than the first threshold, the motion estimation circuit further determines that the first block is a background block.

18. The video processor of claim 13, wherein: The vector processing circuit selects the modified motion vector for the first block from a plurality of candidate motion vectors for a plurality of neighboring blocks around the first block.

19. The video processor of claim 18, wherein: The modified motion vector is selected from one of the plurality of candidate motion vectors, and the selected candidate motion vector is used for an adjacent block among the plurality of adjacent blocks that is determined to be a background block according to the segmentation map.

20. The video processor of claim 18, wherein: The modified motion vector is selected based on a cost function related to at least one of: a distance between the first block and each of the plurality of neighboring blocks, and a sum of absolute errors of each of the plurality of candidate motion vectors.

21. The video processor of claim 13, wherein: When a first pixel projected by the modified motion vector on the current frame is indicated by a first segmentation mark among the plurality of segmentation marks as having a foreground image, and a second pixel projected by the modified motion vector on the previous frame is indicated by a second segmentation mark among the plurality of segmentation marks as having a background image, the vector processing circuit determines that the first block is located in the coverage area.

22. The video processor of claim 13, wherein: When a first pixel projected by the modified motion vector on the current frame is indicated by a first segmentation mark among the multiple segmentation marks as having a background image, and a second pixel projected by the modified motion vector on the previous frame is indicated by a second segmentation mark among the multiple segmentation marks as having a foreground image, the vector processing circuit determines that the first block is located in the non-encapsulated area.

23. The video processor of claim 13, wherein: The vector processing circuit determines that the first block is located in the covered area, and the motion compensation circuit determines that a third pixel projected by the original motion vector on the current frame has a foreground image or a background image based on an indication of a corresponding third segmentation flag among the multiple segmentation flags, and selects the original motion vector as the final motion vector when the third segmentation flag indicates that the third pixel has the foreground image, or selects the modified motion vector as the final motion vector when the third segmentation flag indicates that the third pixel has the background image.

24. The video processor of claim 13, wherein: The vector processing circuit determines that the first block is located in the non-encapsulated area, and the motion compensation circuit determines whether a fourth pixel projected by the original motion vector on the previous frame has a foreground image or a background image based on an indication of a corresponding fourth segment flag among the multiple segment flags, and selects the original motion vector as the final motion vector when the fourth segment flag indicates that the fourth pixel has the foreground image, or selects the modified motion vector as the final motion vector when the fourth segment flag indicates that the fourth pixel has the background image.

25. A video processor, characterized in that: The video processor is used to receive a segmentation map including a plurality of segmentation marks to generate an interpolated frame having a plurality of blocks according to a previous frame and a current frame, the video processor comprising: a motion estimation circuit for generating an original motion vector for a first block among the plurality of blocks, and according to the plurality of segment flags in a region corresponding to the first block, determining whether to mark the first block as a segment boundary; a vector processing circuit coupled to the motion estimation circuit, for generating a modified motion vector for the first block when the motion estimation circuit marks the first block as the segment boundary; and A motion compensation circuit is coupled to the vector processing circuit and is used for selecting a final motion vector from the original motion vector and the modified motion vector according to the segment map for a specific pixel of the first block.

Images