Shielding area detection and restoration method based on motion information
By detecting the consistency of forward and backward motion vectors of image blocks, and combining multiple iterations and weighted fusion, the problem of inaccurate motion vector estimation in occluded areas is solved, enabling accurate detection and repair of occluded areas and improving the quality of video frame interpolation images.
Patent Information
- Application Number
- CN202511202401.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-26
- Publication Date
- 2025-12-12
AI Technical Summary
In video processing, occlusion caused by the movement of foreground objects can lead to inaccurate estimation of motion vectors in the occluded areas of the image, resulting in artifacts and ghosting, which affects image quality.
By detecting the consistency of the forward and backward motion vectors of image blocks, and combining multiple rounds of iteration and weighted fusion, occlusion regions are accurately identified. The existence of occlusion regions is determined by analyzing the consistency error function, and multiple rounds of iterative mapping and backtracking operations are used to improve detection accuracy.
It improves the accuracy of occlusion detection, reduces artifacts and ghosting, and enhances the quality of video frame interpolation and the smoothness of dynamic images.
Smart Images
Figure CN121120455A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of video image processing, in particular to a method for detecting and repairing an occlusion region based on motion information. BACKGROUND
[0002] With the increasing demand for high frame rate and high quality in video applications, the role of image estimation and compensation technology (Motion Estimation and Motion Compensation, MEMC) in video processing is becoming increasingly important. This technology compares the contents of adjacent frames of images, estimates the motion information of pixels or image blocks on the time axis, i.e., motion vectors (MV), and reconstructs the image content based on the obtained motion vectors. Through this process, intermediate frame interpolation can be effectively realized, thereby improving the video frame rate and enhancing the smoothness and visual experience of dynamic pictures. MEMC is widely used in video frame interpolation display, super-resolution reconstruction, video compression encoding, and other fields.
[0003] However, due to the movement of foreground objects, occlusion often occurs in images, i.e., some regions are visible in the previous frame but are covered (Cover) by moving objects in the current frame, or are not visible in the previous frame but are uncovered (Uncover) by moving objects in the current frame. Such occlusion and uncovering regions caused by object displacement are collectively referred to as occlusion regions (OCC). In the OCC region, due to the lack of matching image content between the current frame and the previous frame, the estimation of motion vectors is often inaccurate, resulting in artifacts and residual images in motion compensation, which seriously affects the image quality.
[0004] Therefore, accurately detecting the OCC region and restoring the MV of the region is the key to further improving the quality of the interpolated image. The present application proposes a method for detecting and repairing an occlusion region based on motion information to solve the above problems. SUMMARY
[0005] The present application aims to provide a method for detecting and repairing an occlusion region based on motion information to solve the problems mentioned in the background art. The main idea is that the occlusion region lacks matching image content between adjacent frames, resulting in significant inconsistencies (i.e., direction and amplitude) in the forward motion vector (Forward Motion Vector, ) and the backward motion vector (Backward Motion Vector, ) of adjacent frames, thereby accurately detecting the region based on the inconsistencies and introducing iteration and weighted fusion of consistency calculation.
[0006] The present application provides The detection method takes image blocks as the basic unit, that is, each image block corresponds to one , and combines the forward and backward motion vectors between the previous frame (previous frame, ) and the current frame (current frame, ) for processing, including Cover detection and Uncover detection two stages, as follows are the specific steps: including Cover detection and Uncover detection two stages, the specific implementation logic of the Cover detection and Uncover detection two stages is as follows: First stage: detecting the Cover area: taking the previous frame image as the current frame, and the adjacent frame image as the reference frame, performing Cover area detection; the Cover area feature is: the Cover area exists in , but is blocked by other objects in ; the Cover area detection steps are as follows: Step 1, mapping and backtracking: for the current processing image block of the current frame , set the forward of the image block as the starting , denoted as , map the image block to the corresponding position of the reference frame by using , and extract the backward of the most adjacent image block at the position, denoted as the mapping , that is ; then, map to the current frame in reverse, extract the forward of the most adjacent image block at the mapping position, denoted as the backtracking , that is ; record , and as the basis for subsequent consistency analysis, wherein indicates the current round of mapping and backtracking; Step 2, consistency determination and calculation: by analyzing the consistency relationship between the starting , mapping and backtracking obtained in step 2, it is determined whether the current processing image block belongs to the area, which is specifically determined based on the consistency error function analysis through the normalized consistency error function, which is defined as follows: ; , where is the maximum value of the consistency error function, and the numerical value is equal to the displacement of 64 pixel points ; is the maximum value of the consistency error function, and the numerical value is equal to the displacement of 64 pixel points ; ; , where is the gain parameter, and is set to 0.25 is the minimum value of the consistency error threshold, and the size is the displacement of 8 pixel points ; if the value is equal to 0, it means that the vectors are completely consistent; if it is equal to 1, it means that they are completely inconsistent; based on the analysis of the consistency error function, the following judgment basis can be obtained: The starting , mapping and backtracking of each iteration are analyzed using the consistency error function to determine whether the current processing image block meets the condition, where the condition is: If the starting and the mapping consistency is high, the backtracking of the current processing image block at the corresponding position in the reference frame is mutually referenced, the content of the current processing image block matches the current frame, and the current processing image block is determined to be a non region; if the starting and the mapping consistency is low, it means that the consistency calculation result is greater than the set threshold, the image content of the region only exists in the current frame, and since the reference frame does not have the image content of the region, the content of the current processing image block does not match the current frame, and the current processing image block is a region; If the backtracking and the mapping consistency is high, it means that the mapping and the backtracking of the reference frame are mutually referenced, the content of the current processing image block matches the current frame, and it means that the current processing image block does not belong to the region; if the backtracking and the mapping consistency is low, it cannot be determined whether it is a region, and a penalty term is added in the next iteration calculation to make the result of the next calculation more biased towards the region; If the starting and the backtracking Low consistency; after mapping and backtracking, the motion vectors are unstable, indicating that the currently processed image block is... Region; if starting With back lookup High consistency indicates that the currently processed image patch maintains a stable direction and amplitude of motion during the mapping and lookup process, suggesting that the currently processed image patch does not belong to... area; Step 3, Multi-round Iterative Mapping and Backtracking: After completing steps 11 and 12, perform multi-round iterative mapping and backtracking on the current image block, and then perform consistency determination and... Score calculation; specifically, let's say we are performing the first... Wheel mapping and lookup, then the starting point of that wheel. For the previous round of backtracking Repeat step 11 to obtain the first... The beginning of the wheel Mapping Retrospective review Repeat step 2 to calculate consistency and... Score; after Iterative mapping before and after the round enhances stability and accuracy; Step 4, OCC Score Weighted Fusion: Due to the influence of MV accuracy or deviations in forward and backward motion estimation, OCC region edges or moving object edges in the image are prone to missed detections and false detections. The scores calculated in all rounds of Step 2 are weighted and fused together. The results were obtained through weighted fusion. Scoring results can effectively reduce false positives and false negatives, making the results more accurate. Phase Two: Detecting the Uncovered Area: Uncovered detection uses... As a reference frame, For the current frame, perform Uncover region detection; the characteristics of the Uncover region are: the Cover region is in It was obscured in the middle, but The above is revealed due to the movement of the object; The Uncover region detection logic is as follows: use the previous frame as the reference frame. Adjacent frame images are used as the current frame. For the current frame The currently processed image block, let the backward direction of this image block be... Starting , recorded as ,use Map the image patch to the reference frame The corresponding position, and extract the forward image of the nearest image patch at that position. , denoted as mapping i.e. ; then, the backward mapping to the current frame , the backward of the image block nearest to the mapping position is extracted, denoted as look back i.e. ; then, steps 2-4 are repeated to obtain score results.
[0007] Preferably, for judging whether the image block to be processed meets the condition, wherein the score calculation can be expressed by the above relationship in a formula, as follows: ; In the formula, represents the direction conversion from the current frame to the reference frame, is a determination threshold, which is set according to the accuracy and the numerical value of ; is a calculation gain, which is set to 1.5; represents a penalty term, which is defined as follows: ； In the formula, represents a penalty term gain, which is set to 0.1; , the numerical value of which represents the possibility of the image block being a region in this round of calculation; when , the image block is determined to be a non- region in this round; when , the image block is determined to be a region in this round.
[0008] Preferably, the OCC results calculated in all rounds are fused by weighting to obtain score results, and the specific calculation formula is as follows: ; In the formula, is a threshold, which is set to 0.2; is a gain, which is set to 1.2; represents the weight of the round result, which satisfies the following requirements: , The setting of increases with the increase of the value.
[0009] Compared with the prior art, the beneficial effects of the present invention are: 1. Improve the accuracy of occluded area detection: Based on motion vector consistency analysis: By analyzing the differences in direction and amplitude of forward and backward motion vectors, Cover (visible in the previous frame but occluded in the current frame) and Uncover (occluded in the previous frame but visible in the current frame) regions are accurately identified, thus solving the problem of motion vector estimation distortion caused by missing image content in traditional methods.
[0010] Enhanced stability through multiple iterations: By iterating and backtracking 2-4 times, the consistency judgment of motion vectors is gradually optimized, reducing the randomness error of a single calculation and making the detection of occluded area boundaries more accurate.
[0011] Weighted fusion reduces false positives and false negatives: for multi-round iterations The scores are weighted and merged (e.g., weights increase with each round: 0.2, 0.3, 0.5), combined with dynamic threshold adjustments, to effectively avoid [the problem caused by] [the following]. The problem of missed or false detections in edge areas caused by accuracy deviation or motion estimation error.
[0012] 2. Improve the quality of video frame interpolation images: Reduce artifacts and ghosting: After accurately detecting occluded areas, motion vector recovery in unoccluded areas can be optimized in a targeted manner, avoiding artifacts and ghosting issues that occur in traditional MEMC technology due to occlusion. The distortion caused by frame interpolation artifacts and ghosting improves the smoothness of dynamic images. Attached Figure Description
[0013] Fig. 1 For the present invention Schematic diagram of the testing process structure; Fig. 2 This is a schematic diagram illustrating the mapping and lookup principle of the present invention. Detailed Implementation
[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0015] Please see Figs. 1-2 This invention provides a technical solution: a method for detecting and repairing occlusion areas based on motion information, comprising two stages: Cover detection and Uncover detection. The specific implementation logic of the Cover detection and Uncover detection stages is as follows: First stage: detecting Cover region: taking the last frame image as the current frame, and the adjacent frame image as the reference frame, performing Cover region detection; Cover region feature is: Cover region exists in , but is blocked by other objects in ; Cover region detection steps are as follows: Step 1, mapping and backtracking: for the current processing image block of the current frame , set the forward of the image block as the starting , denoted as , map the image block to the corresponding position of the reference frame by using , and extract the backward of the most adjacent image block at the position, denoted as the mapping , that is ; then, map to the current frame in reverse, extract the forward of the most adjacent image block at the mapping position, denoted as the backtracking , that is ; record , and as the basis for subsequent consistency analysis, wherein represents the current round of mapping and backtracking, and the iteration number is 2-4 times (usually 3 times), which controls the amount of calculation while ensuring detection accuracy, is more engineering practical than full-frame dense calculation, and is suitable for real-time video processing system integration; Step 2, consistency determination and calculation: by analyzing the consistency relationship between the starting , mapping and backtracking obtained in step 2, determine whether the current processing image block belongs to region, which is specifically determined based on consistency error function analysis through a normalized consistency error function, defined as follows: ; wherein is the maximum value of the consistency error function, and the numerical size is equal to the displacement of 64 pixel points; and are any two motion vectors, is a consistency error threshold, which is defined as follows: ; wherein, is a gain parameter, set to 0.25; is a minimum value of consistency error threshold, with a size of 8 pixels displacement; if value is equal to 0, it means that the vectors are completely consistent; if equal to 1, it means that they are completely inconsistent; based on the analysis of the consistency error function, the following judgment basis can be obtained: the starting , mapping and backtracking of each iteration are analyzed using the consistency error function to determine whether the current processing image block meets condition, wherein the condition is: if the starting and the mapping consistency is high, the backtracking of the current processing image block in the corresponding position of the reference frame is mutually referenced, the content of the current processing image block matches the current frame, and the current processing image block is determined to be a non region; if the starting and the mapping consistency is low, it means that the consistency calculation result is greater than the set threshold, the image content of the region only exists in the current frame, and since the reference frame does not have the image content of the region, the content of the current processing image block does not match the current frame, and the current processing image block is a region; if the backtracking and the mapping consistency is high, it means that the mapping and the backtracking are mutually referenced, the content of the current processing image block matches the current frame, and it means that the current processing image block does not belong to the region; if the backtracking and the mapping consistency is low, it cannot be determined whether it is a region, and a penalty term is added in the next iteration calculation to make the calculation result of the next round more biased towards the region; if the starting and the backtracking consistency is low, the motion vector is unstable after mapping and backtracking, which means that the current processing image block is a region; if the starting and the backtracking consistency is high, it means that the current processing image block maintains a stable motion direction and amplitude during the mapping and backtracking process, which means that the current processing image block does not belong to the region; for determining whether the to-be-processed image block meets Conditions, among which The score calculation can be expressed using the above relationship as a formula, as follows: ; In its formula, express After the direction conversion from the current frame to the reference frame... The threshold is set according to... Adjustment of precision and numerical value; for Calculate the gain and set it to 1.5; The penalty term is defined as follows: ； In its formula, This represents the penalty term gain, set to 0.1; Its numerical value indicates the value of the image patch in this round of calculation. The possibility of the region; when The image block was determined to be non-in this round. Area; when The image block was determined to be in this round. area; Step 3, Multi-round Iterative Mapping and Backtracking: After completing steps 11 and 12, perform multi-round iterative mapping and backtracking on the current image block, and then perform consistency determination and... Score calculation; specifically, let's say we are performing the first... Wheel mapping and lookup, then the starting point of that wheel. For the previous round of backtracking Repeat step 11 to obtain the first... The beginning of the wheel Mapping Retrospective review Repeat step 2 to calculate consistency and... Score; after Iterative mapping before and after the round enhances stability and accuracy; Step 4, OCC Score Weighted Fusion: Due to the influence of MV accuracy or deviations in forward and backward motion estimation, OCC region edges or moving object edges in the image are prone to missed detections and false detections. The scores calculated in all rounds of Step 2 are weighted and fused together. The results were obtained through weighted fusion. Scoring results can effectively reduce false positives and false negatives, making the results more accurate. The OCC results calculated in all rounds are then weighted and fused to obtain... The scoring results are calculated using the following formula: ; In its formula, for The threshold is set to 0.2; for The gain is set to 1.2. Indicates the first wheel The weights of the results shall satisfy the following requirements: , The settings are as follows The value increases as it increases.
[0016] Phase Two: Detecting the Uncovered Area: Uncovered detection uses... As a reference frame, For the current frame, perform Uncover region detection; the characteristics of the Uncover region are: the Cover region is in It was obscured in the middle, but The above is revealed due to the movement of the object; The Uncover region detection logic is as follows: use the previous frame as the reference frame. Adjacent frame images are used as the current frame. For the current frame The currently processed image block, let the backward direction of this image block be... Starting , recorded as ,use Map the image patch to the reference frame The corresponding position, and extract the forward image of the nearest image patch at that position. , denoted as mapping ,Right now ; then, Reverse mapping to the current frame Extract the backward image patch of the nearest mapping location. Record as a back lookup ,Right now Then repeat steps 2-4 to obtain... Score results.
[0017] It should be noted that the steps and formulas used in Uncover detection are exactly the same as those in Cover region detection. The only difference is: In the round of iteration, the initial image patch For backward ,Right now ; mapping Forward ,Right now ; Retrospective check For the reverse , i.e. .
[0018] The application provides a method for detecting and repairing occlusion area based on motion information. The method determines Cover and Uncover area based on inter-frame motion estimation, and uses inconsistency of forward motion vector and backward motion vector at image block level. The method takes image block as basic processing unit, and image block size can be selected as 8x8 or 16x16 pixels, The minimum precision can be selected as 0.25 pixel points, i.e. moving 0.25 pixel points in horizontal coordinate corresponds to 1, through multiple rounds of iteration mapping and backtracking operation, combining with consistency measurement function, the accuracy and robustness of detection are improved.
[0019] Although the embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made thereto without departing from the principles and spirit of the present application, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for detecting and repairing occluded regions based on motion information, characterized in that, The process includes two phases: Cover detection and Uncover detection. The specific implementation logic of these two phases is as follows: Phase 1: Detecting the Cover Region: Using the previous frame image Considered as the current frame, adjacent frame images Treating it as a reference frame, perform Cover region detection; the Cover region feature is: the Cover region is in It exists in, but The Cover region detection steps are as follows: (The area is occluded by other objects.) Step 1, Mapping and Lookup: For the current frame The currently processed image block, let the forward direction of this image block be... Starting , recorded as ,use Map the image patch to the reference frame The corresponding position is determined, and the backward image of the nearest image patch at that position is extracted. , denoted as mapping ,Right now ; then, Reverse mapping to the current frame Extract the forward image patch that is closest to the mapped location. Record as a back lookup ,Right now ;Record , and As the basis for subsequent consistency analysis, among which Indicates the current round of mapping and lookup; Step 2, Consistency Determination and Calculation: Based on the initial result obtained in step 2... Mapping and backtracking The consistency relationship between them determines whether the currently processed image patch belongs to [the specified category]. The region is specifically determined based on the consistency error function analysis, which is performed using a normalized consistency error function, defined as follows: ； In its formula, The maximum value of the consistency error function is equal to the value of 64 pixels. Displacement; and For any two motion vectors, The consistency error threshold is defined as follows: ； In its formula, The gain parameter is set to 0.25; The minimum value of the consistency error threshold, with a size of 8 pixels. Displacement; if A value of 0 indicates that the vectors are completely consistent; a value of 1 indicates that they are completely inconsistent. Based on the analysis of the consistency error function, the following judgment criteria can be obtained: Analyzing the beginning of each iteration using the consistency error function Mapping and back check Determine whether the currently processed image block satisfies Conditions, among which the following conditions are met The conditions are: If the beginning With mapping High consistency; the motion of the currently processed image patch in the current frame is compared with the corresponding position in the reference frame. Using each other as a reference, the content of the currently processed image block is matched with the current frame to determine if the currently processed image block is non-standard. Region; if starting With mapping Low consistency means that the consistency calculation result is greater than the set threshold. The region's image content exists only in the current frame. Since the reference frame does not contain image content for that region, meaning the content of the currently processed image block does not match the current frame, the currently processed image block is... area; If you need to check back With mapping High consistency indicates a good mapping of the reference frame. and backtracking If the currently processed image patch matches the content of the current frame, it means that the currently processed image patch does not belong to the current frame. Area; if you need to check back With mapping Low consistency, unable to determine whether it is The region is used to add a penalty term to the next iteration of the calculation, making the result of the next round of calculation more biased towards [the region]. area; If the beginning With back lookup Low consistency; after mapping and backtracking, the motion vectors are unstable, indicating that the currently processed image block is... Region; if starting With back lookup High consistency indicates that the currently processed image patch maintains a stable direction and amplitude of motion during the mapping and lookup process, suggesting that the currently processed image patch does not belong to... area; Step 3, Multi-round Iterative Mapping and Backtracking: After completing steps 11 and 12, perform multi-round iterative mapping and backtracking on the current image block, and then perform consistency determination and... Score calculation; specifically, let's say we are performing the first... Wheel mapping and lookup, then the starting point of that wheel. For the previous round of backtracking Repeat step 11 to obtain the first... The beginning of the wheel Mapping Retrospective review Repeat step 2 to calculate consistency and... Score; after Iterative mapping before and after the round enhances stability and accuracy; Step 4: Calculate the results for all rounds in Step 2. The results were obtained through weighted fusion. Score results; Phase Two: Detecting the Uncovered Area: Uncovered detection uses... As a reference frame, For the current frame, perform Uncover region detection; the characteristics of the Uncover region are: the Cover region is in It was obscured in the middle, but The above is revealed due to the movement of the object; The Uncover region detection logic is as follows: use the previous frame as the reference frame. Adjacent frame images are used as the current frame. For the current frame The currently processed image block, let the backward direction of this image block be... Starting , recorded as ,use Map the image patch to the reference frame The corresponding position, and extract the forward image of the nearest image patch at that position. , denoted as mapping ,Right now ; then, Reverse mapping to the current frame Extract the backward image patch of the nearest mapping location. Record as a back lookup ,Right now Then repeat steps 2-4 to obtain... Score results.
2. The method for detecting and repairing occlusion areas based on motion information according to claim 1, characterized in that, To determine whether the image block to be processed satisfies Conditions, among which The score calculation can be expressed using the above relationship as a formula, as follows: ； In its formula, express After the direction conversion from the current frame to the reference frame... The threshold is set according to [the following criteria]. Adjustment of precision and numerical value; for Calculate the gain and set it to 1.5; The penalty term is defined as follows: ； In its formula, This represents the penalty term gain, set to 0.1; Its numerical value indicates the value of the image patch in this round of calculation. The possibility of the region; when The image block was determined to be non-in this round. Area; when The image block was determined to be in this round. area.
3. The method for detecting and repairing occlusion areas based on motion information according to claim 1, characterized in that, The OCC results calculated in all rounds are then weighted and fused to obtain... The scoring results are calculated using the following formula: ； In its formula, for The threshold is set to 0.2; for The gain is set to 1.
2. Indicates the first wheel The weights of the results shall satisfy the following requirements: , The settings are as follows The value increases as it increases.