Video Processing Method for Correcting Motion Vectors

Abstract

A video processing method includes dividing a current frame into multiple blocks, generating a motion vector for each block of the current frame according to each block of the current frame and a corresponding block of a previous frame, generating a global motion vector according to the multiple motion vectors of the current frame, generating a pixel absolute error sum for each block of the current frame according to the global motion vector, generating a region including multiple blocks of the current frame, matching the distribution of the multiple pixel absolute error sums of the region with multiple models, finding the model that best matches the distribution of the multiple pixel absolute error sums of the region, and labeling each block of the region as a foreground or background according to the best matching model.

Description

Technical Field

[0001] The present invention relates to a video processing method, and more particularly to a video processing method capable of correcting motion vectors. Background Art

[0002] Motion Estimate and Motion Compensation (MEMC) can be used in the field of video processing to convert low frame rate to high frame rate. The accuracy of the motion vector calculated by the motion estimate will determine the quality of the inserted intermediate frames and affect the quality of the final high frame rate video.

[0003] The existing motion prediction method usually cuts the image into a number of small macroblocks and calculates the motion vector for each macroblock. Since the comparison based on macroblocks is unstable, the motion vector generated by the mutual reference between macroblocks will inevitably cause the motion vector mismatch at the edge of the object in the video and the intersection of the foreground and the background. In other words, the motion vector of the background will appear in the foreground object, and the motion vector of the foreground object will appear in the background, which will affect the final motion compensation effect and thus fail to improve the quality of the final high frame rate video. Summary of the invention

[0004] The present invention application provides a video processing method, which includes dividing a current frame into multiple blocks, generating a motion vector for each block of the current frame according to each block of the current frame and a corresponding block of a previous frame, generating a global motion vector according to the multiple motion vectors of the current frame, generating a pixel sum of absolute difference (SAD) for each block of the current frame according to the global motion vector, generating a region including the multiple blocks of the current frame, matching the distribution of the multiple pixel sums of absolute difference of the region with multiple models, finding the model that best matches the distribution of the multiple pixel sums of absolute difference of the region, and labeling each block of the region as a foreground or background according to the best matching model.

[0005] The present invention application also provides a video processing system, comprising a processor and a memory, wherein the processor divides the current frame of the video into multiple blocks, generates a motion vector for each block of the current frame based on each block of the current frame and the corresponding block of the previous frame, generates a global motion vector based on the multiple motion vectors of the current frame, generates a pixel absolute error sum for each block of the current frame based on the global motion vector, generates a region including the multiple blocks of the current frame, matches the distribution of the multiple pixel absolute error sums of the region with multiple models, finds the model that best matches the distribution of the multiple pixel absolute error sums of the region, and labels each block of the region as foreground or background based on the best matching model. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figures 1A to 1C FIG. 4 is a schematic diagram of a video frame according to an embodiment of the present invention.

[0007] Figure 2A FIG. 4 is a schematic diagram of video frame segmentation according to an embodiment of the present invention.

[0008] Figure 2B FIG. 4 is a schematic diagram of a video frame motion vector according to an embodiment of the present invention.

[0009] Figure 3 FIG. 4 is a schematic diagram of movement of a portion of video frames according to an embodiment.

[0010] Figure 4A Schematic diagram of a model of pixel absolute error and distribution in an embodiment.

[0011] Figure 4B Schematic diagram of the video frame matching area in the embodiment.

[0012] Figure 5 is a flow chart of a video processing method.

[0013] Figure 6 A schematic diagram of a computer system for executing a video processing method.

[0014] Explanation of symbols:

[0015] F0, F1: video frame

[0016] Obj: foreground object

[0017] Bac: Background

[0018] X, Y: Block

[0019] A~D:Model

[0020] R: Region

[0021] 500: Methods

[0022] S502~S516: Steps

[0023] 600: Computer Systems

[0024] 610: Processor

[0025] 620: Memory DETAILED DESCRIPTION

[0026] The embodiments will be described in detail below in conjunction with the accompanying drawings, but the specific embodiments described are only used to explain the present invention and are not used to limit the application of the present invention. The description of the operations is not used to limit the order of their execution. Any recombination to produce a method or system with equivalent efficacy falls within the scope covered by the content of the application of the present invention.

[0027] Figures 1A to 1C Schematic diagram of a video frame of an embodiment of the present invention. The previous frame F0 is the picture of the video at time T-1, and the current frame F1 is the picture of the video at time T. In the video, the foreground object obj moves to the right with the background bac as a reference; the background bac moves to the left with respect to the foreground object obj.

[0028] Figure 2A FIG. 4 is a schematic diagram of video frame segmentation according to an embodiment of the present invention. Figure 2B FIG. 1 is a schematic diagram of a video frame motion vector according to an embodiment of the present invention. Figure 2A As shown, using the conventional motion compensation technology, the current frame F1 is divided into several blocks, each block consists of M×M pixels, and M can be a positive integer. Then, according to each block in the current frame F1, the corresponding block is found in the previous frame F0, so that the motion vector of the motion direction and size of each block in the current frame F1 can be calculated, as shown in FIG. Figure 2B As shown. Then, according to the motion vector (MV) of each block, the global motion vector (GMV) of the overall motion direction and size of the picture is calculated. Generally speaking, the block matching algorithm is most commonly used to find the motion vector. Each frame of the dynamic image is cut into blocks of equal size, and the current block in the picture has a corresponding search area in the previous frame. In the corresponding search area, find the block that is most similar to the current block, that is, the block with the smallest error, and then record the motion vector.

[0029] Figure 3 FIG. 1 is a schematic diagram of the movement of some video frames in an embodiment. The figure shows video frames at time T-1, T and T+1. Figure 3As shown in block X, since block X only contains background bac, the sum of absolute differences (SAD) calculated using the global motion vector will be relatively small; Figure 3 As shown in block Y, since block Y contains the foreground object obj in addition to the background bac, the calculated pixel absolute error sum will be larger. At the edge of the foreground object obj, the pixel absolute error sum calculated by the global motion vector will change from small to large.

[0030] The SAD is a measure of similarity between blocks, obtained by calculating the absolute difference between the pixels of two corresponding blocks. The smaller the SAD, the more similar the blocks are. The calculation formula of the SAD is as follows:

[0031]

[0032] Where a(i,j) represents the pixel in the i-th column and j-th row of the current block of size M×M, b(i+u,j+v) represents the pixel in the i+u-th column and j+v-th row of the corresponding search area, and (u,v) represents the motion vector. The meaning of the entire calculation formula is the accumulation of the absolute error values ​​of the current block and the candidate block with the motion vector (u,v) in the search area. The smaller the value of the absolute error sum is, the more similar the candidate block is to the current block.

[0033] In the embodiment, the global motion vector is substituted into the motion vector (u, v) to obtain the pixel absolute error sum calculated by the global motion vector. By using this method, the absolute error sum of each block in the current frame F1 can be obtained.

[0034] Figure 4A Schematic diagram of pixel absolute error and distribution model in the embodiment. Figure 4B Schematic diagram of the video frame matching area in the embodiment. Figure 4A Figure 2 shows four pixel absolute error and distribution models of model A, model B, model C, and model D. These default models contain N×N blocks, where N is a positive integer, such as 2, 5, or 10. Each block of model A to D has a label. If the pixel absolute error of the block is less than a threshold, the label of the block is background; if the pixel absolute error of the block is greater than a threshold, the label of the block is foreground.

[0035] Take an area of ​​the same size as the default model from the video frame, that is, an area containing N×N blocks, and match it with the model's pixel absolute difference (SAD). The number of matched blocks is used to determine which model the area belongs to. Figure 4B Take a 2×2 block area R from the video frame. Compared with models A to D, area R has the most absolute pixel errors and matches between the blocks and model C. In this way, it can be determined that area R matches model C. After the best matching model is determined, such as model C, each block in area R can be labeled as foreground or background according to the foreground or background label of the best matching model C. Repeat the above steps until every block in the video frame is labeled.

[0036] Finally, the motion compensation technology can be used to correct the motion vector of each block according to the label of each block, thereby improving the accuracy of the motion vector at the junction of the moving object and the background, thereby improving the integrity of the object edge when using motion compensation.

[0037] Figure 5 is a flowchart of a video processing method 500, wherein the video processing method 500 comprises the following steps:

[0038] S502: Divide the current frame into multiple blocks;

[0039] S504: Generate a motion vector of the block according to each block of the current frame and the corresponding block of the previous frame;

[0040] S506: Calculate a global motion vector based on multiple motion vectors of the current frame;

[0041] S508: Generate a pixel absolute error sum of each block according to the global motion vector;

[0042] S510: Generate a region including a plurality of blocks in the current frame;

[0043] S512: Match the distribution of the sum of absolute pixel errors of the blocks in the region with the model, and find the model that best matches the distribution of the sum of absolute pixel errors;

[0044] S514: Label each block of the region as foreground or background according to the best matching model; and

[0045] S516: Correct the motion vector of each block according to the label of each block.

[0046] Figure 6Schematic diagram of a computer system 600 for executing the video processing method 500. The computer system 600 includes a processor 610 and a memory 620. The memory 620 stores a computer program for executing the video processing method 500 via the processor 610. For example, the memory 620 may include, for example, a flash memory, a disk, such as an internal hard disk or a removable disk. The processor and the memory may be supplemented by an application specific logic circuit (ASIC) or directly incorporated into the application specific logic circuit. The processor 610 may be a processor suitable for executing a computer program, including general-purpose and special-purpose microprocessors. The computer program may be written in any form of programming language.

[0047] This method aims to solve the problem that motion compensation is often inaccurate at the edge of moving objects. It uses a simple, fast and relatively accurate method to identify the foreground and background areas, separate the moving objects in the picture, and correct the motion vector at the junction of the moving object and the background to improve the accuracy of the motion vector. The method of using multiple models for matching can be close to the slope of the object edge to a certain extent, improving the integrity of the object edge when using motion compensation.

[0048] Although the content of the present invention application has been disclosed as above through specific implementation methods, it is not intended to limit the content of the present invention application. Ordinary technicians in this field should be able to make various modifications or adjustments without departing from the concept and scope of the content of the present invention application. Therefore, the protection scope of the content of the present invention application shall be based on the scope defined in the claims.

Claims

1. A video processing method, It is characterized in that The video processing method comprises: Divide the current frame into multiple blocks; Generate a motion vector for each block of the current frame according to each block of the current frame and a corresponding block of a previous frame; generating a global motion vector according to the multiple motion vectors of the current frame; Generating a pixel absolute error sum of each block of the current frame according to the global motion vector; generating a region including a plurality of blocks of the current frame; Matching the distribution of the sum of absolute errors of the multiple pixels in the region with multiple models, and finding the model that best matches the distribution of the sum of absolute errors of the multiple pixels in the region; and Labeling each block of the region as foreground or background according to the best matching model; The video processing method further includes modifying the motion vector of each block of the current frame according to the label of each block of the current frame.

2. The method according to claim 1, It is characterized in that Each block of the current frame includes M×M pixels, where M is a positive integer.

3. The method according to claim 1, It is characterized in that The plurality of blocks of the region are N×N blocks, where N is a positive integer.

4. The method according to claim 1, It is characterized in that Each block of the multiple models has a label, and if the sum of absolute pixel errors of the blocks of the multiple models is less than a threshold, the block label is background.

5. A video processing system, It is characterized in that The video processing system comprises: A processor to process the video; and A memory, coupled to the processor, for temporarily storing the video; wherein the processor divides a current frame of the video into a plurality of blocks, generates a motion vector for each block of the current frame according to each block of the current frame and a corresponding block of a previous frame, generates a global motion vector according to the plurality of motion vectors of the current frame, generates a pixel absolute error sum of each block of the current frame according to the global motion vector, generates a region including the plurality of blocks of the current frame, matches the distribution of the plurality of pixel absolute error sums of the region with a plurality of models, finds a model that best matches the distribution of the plurality of pixel absolute error sums of the region, and labels each block of the region as a foreground or a background according to the best matching model; The processor modifies the motion vector of each block of the current frame according to the label of each block of the current frame.

6. The video processing system according to claim 5, It is characterized in that Each block of the current frame includes M×M pixels, and M is a positive integer.

7. The video processing system according to claim 5, It is characterized in that The plurality of blocks of the area is N×N blocks, and N is a positive integer.

8. The video processing system according to claim 5, It is characterized in that Each block of the multiple models has a label, and if the sum of absolute pixel errors of the blocks of the multiple models is less than a threshold, the block label is background.

Images