Fast Block Partitioning Method for Multi-view Video Coding

Abstract

The present invention discloses a fast block partitioning method for multi-view video coding, comprising: S1, reducing the resolution of the coding frame in the coding sequence to 1 / 2 of the original, and performing coarse intra-frame prediction and coarse inter-frame prediction in units of 16*16 pixels; S2, using the SIFT feature extraction algorithm to calculate the average offset vector V between the reference viewpoint and the referenced viewpoint. offset (x, y); S3, after the coding of the reference viewpoint is completed, the depth map of the reference viewpoint is stored according to the division structure of each coding tree unit CTU of the reference viewpoint; S4, for each CTU of the reference viewpoint, before actual coding, according to its position (x, y) in the picture and the obtained average offset vector V offset (x, y), construct the depth region DR based on the current block to be encoded; S5, place the constructed DR in the corresponding position of the reference viewpoint, and summarize the depth distribution of the depth map in the DR; S6, encode the reference viewpoint coding block; S7, repeat the above steps S4 to S6 for the coding blocks of all reference viewpoints until all coding blocks of the current viewpoint are encoded.

Description

Technical Field

[0001] The present invention belongs to the technical field of video coding, and in particular relates to a fast block division method for multi-viewpoint video coding. Background Art

[0002] With the rapid development of multimedia technology in recent years, people are no longer satisfied with simple high-definition video applications. Instead, they desire to receive and experience video content from multiple perspectives through their senses. This demand has driven the evolution of standard single-viewpoint video content applications towards multi-viewpoint video applications. Multi-viewpoint video refers to video captured simultaneously by multiple cameras from different angles. In this type of video application, multi-viewpoint video compression is a key research and component. International standards groups have expanded upon single-viewpoint compression standards and released standards for multi-viewpoint video. Currently, the most advanced is the MV-HEVC standard, an extension of the HEVC standard. Compared to simulcast encoding, the technology used in the MV-HEVC standard can reduce the bitrate by 20% to 30% while maintaining equivalent image quality.

[0003] The prediction technology used by MV-HEVC during predictive coding is basically the same as that of HEVC, including 35 angle predictions for intra-frame prediction and AMVP, Merge, and Skip modes for inter-frame prediction. Similarly, during the rate-distortion optimization process, the CTU (Coding Tree Unit) needs to be recursively divided into 64×64, 32×32, 16×16, and 8×8. However, the increase in viewpoints to be compressed also brings a more complex coding process. In addition to the intra-frame prediction and inter-frame prediction of the newly added viewpoint itself, different viewpoints in the same POC (Picture Order Count) will use a new predictive coding method - inter-view prediction to pursue smaller residuals. The principle of inter-view prediction is basically the same as that of inter-frame prediction, and the main difference lies in the different reference frames. The reference frame of inter-frame prediction is the adjacent frame in the time domain, and the motion estimation performed on this frame is called ME (Motion Estimation). The reference frame for inter-view prediction is the adjacent viewpoint frame at the same moment, and the motion estimation performed on this frame is called disparity estimation (DE). The newly added mode also requires the recursive block partitioning process to execute more prediction modes to achieve the best encoding quality, but this makes multi-view encoding much more complex than encoding different viewpoint sequences separately using conventional coding standards (such as HEVC).

[0004] Actual MV-HEVC bitstreams show that the reference view and the referenced view typically have a high degree of similarity. The reference view is the viewpoint used to reduce prediction residuals during inter-view prediction, and in binocular video, this is typically the auxiliary viewpoint. The referenced viewpoint is used as a reference for inter-view prediction, and in binocular video, this is typically the primary viewpoint. The reference viewpoint partitioning structure rarely includes coding blocks with higher depths. The proportion of coding blocks that are not partitioned is as high as 76%, and the proportion of coding blocks that are partitioned only once is 17%. The remaining 7% are coding blocks with more detailed partitioning. Without any fast algorithm optimization, these coding blocks that do not need to be partitioned or only need to be partitioned once still require a complex rate-distortion optimization process, trying multiple prediction modes to ultimately achieve the optimal result. Taking a 64×64 CU as an example, if each recursive partitioning is simplified to perform intra-frame prediction, inter-frame prediction, and inter-view prediction, the CU will need to perform a total of 3×(1+4+16+64)=255 prediction modes. If the optimal mode of the CU is not to split, the execution of the remaining 255-3=252 prediction modes is redundant, wasting a lot of time. Summary of the Invention

[0005] In view of the above problems, the present invention provides a fast block partitioning method for multi-viewpoint video coding, which is used to utilize the correlation between different viewpoints and the partitioning structure of the encoded viewpoints, set a depth threshold for the unencoded viewpoints, and skip some unnecessary partitions, thereby reducing the coding complexity.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A fast block partitioning method for multi-view video coding comprises the following steps:

[0008] S1, based on the pre-analysis lookahead process, reduces the resolution of the coded frame in the sequence to be coded to 1 / 2 of the original, and performs coarse intra-frame prediction and coarse inter-frame prediction in units of 16*16 pixels, and stores the costs obtained from the two predictions in the data structure of the corresponding frame;

[0009] S2, take all viewpoint images of the first ten frames of the video sequence, perform four-fold downsampling, use the SIFT feature extraction algorithm to calculate the average offset vector V between the reference viewpoint and the referenced viewpoint offset (x,y);

[0010] S3, after the encoding of the reference viewpoint is completed, storing the depth map of the reference viewpoint according to the division structure of each coding tree unit CTU of the reference viewpoint, and the storage unit of the depth map is the minimum CU;

[0011] S4, for each CTU of the reference viewpoint, before actual encoding, according to its position (x, y) in the picture and the obtained average offset vector V offset (x, y), build the depth region DR based on the current block to be encoded;

[0012] S5, placing the constructed DR in the corresponding position of the reference viewpoint, summarizing the depth distribution of the depth map in the DR, and taking the depth with the highest proportion as the maximum depth of the block to be coded of the reference viewpoint; the block to be coded takes the minimum value of the obtained coarse intra-frame prediction and coarse inter-frame prediction costs as the optimal cost, and adds the optimal cost of other coding blocks with the same depth, and compares the result with the optimal cost of its parent block. If the optimal cost of the parent block is smaller, the original maximum depth remains unchanged; if the optimal cost of the parent block is larger, the original maximum depth is increased by 1; the parent block is the block corresponding to the depth of the current coding block minus 1, and the block to be coded of the reference viewpoint will limit the number of times it is divided according to the maximum depth;

[0013] S6, refer to the viewpoint coding block for coding division. If the maximum division depth of the current coding block is greater than 2, the division depth distribution of the adjacent coded CTU of the current coding block is obtained; if the proportion of coding blocks with a depth less than 2 is greater than 50%, the depth threshold of the current coding block is set to 1, that is, the coding of depths 2 and 3 is skipped;

[0014] S7 , repeating the above steps S4 to S6 for all coding blocks of reference viewpoints until all coding blocks of the current viewpoint are encoded.

[0015] In one possible implementation, the coarse intra-frame prediction refers to calculating only the SATD (Sumof Absolute Transformed Difference) of a small number of prediction modes, including DC, PLANAR, ANGLE5, 10, 15, 20, 25 and 30, a total of 8 modes. Then, the mode with the best SATD is selected, and the SATD calculation of two prediction angles with a step size of 2 is performed. Finally, the best SATD is selected as the cost and its corresponding prediction mode.

[0016] In a possible implementation, the coarse inter-frame prediction refers to performing a motion search in the reduced reference frame to find the best matching block, and calculating a SATD value of the residual as the cost of the coarse inter-frame prediction.

[0017] In a possible implementation, the calculation of the average offset vector V between the reference viewpoint and the referenced viewpoint is performed. offset (x,y) specifically includes:

[0018] Extract feature points, and assume that the feature point set of the two viewpoint images of the kth image is {P1, P2, ... P N} and {Q1,Q2,…Q N Then, the feature points are matched using the matching algorithm to obtain several matching pairs represented as (P i ,Q j ), for each matched feature point, calculate their coordinate offsets Δx and Δy:

[0019] Δx i =Q jx -P ix

[0020] Δy i =Q jy -P iy

[0021] Among them, P ix and P iy is the coordinate of the i-th feature point of the first viewpoint of the image, Q jx and Q jy is the coordinate of the jth feature point of the second viewpoint of the image;

[0022] Then calculate the average offset of all feature points:

[0023]

[0024] Where N is the number of matching pairs. After the average offset of the viewpoints corresponding to all images is calculated, the total average offset can be obtained. Since the above steps are performed on the image under four times downsampling, in order to correspond to the original image, the result of the total average offset needs to be multiplied by 4 to obtain the final V offset (x,y), that is: in:

[0025]

[0026] When the vector component value is negative, it means it is shifting to the left or downward; when it is positive, it means it is shifting to the right or upward. offset The vector value of (x, y) needs to be aligned to the size of the CU, taking an integer multiple of the minimum CU size. Let the vector before alignment be a(x, y):

[0027] V offset (x,y)=a(x,y)+MiniSize CU -a(x,y)%MiniSize CU

[0028] Among them, MiniSize CU Represents the minimum value of the CU height set by the encoder, usually 8.

[0029] In a possible implementation, the position (x, y) of the pixel in the image and the average offset vector V are obtained before actual encoding. offset (x, y), the depth region DR is constructed based on the current block to be coded, specifically including:

[0030] The height H of this area is: H = H CU +|V offsety |

[0031] The width W is: W = W CU +|V offsetx |

[0032] The starting point of the depth area is:

[0033] DR x =x+V offsetx

[0034] DR y =y+V offsety ;

[0035] Among them, H CU and W CU are the height and width of the CU to be encoded in pixels; V offsetx and V offsety The horizontal and vertical component values ​​of the average offset vector are used. According to the starting point of the area and the H and W values, the DR can be constructed.

[0036] The use of the present invention has the following beneficial effects: existing multi-view video compression standards, such as MV-HEVC, have added new viewpoints, resulting in more prediction modes to be executed during the encoding process, which makes the rate-distortion optimization process more time-consuming, thereby increasing the complexity of the overall encoding, making multi-view video compression difficult to apply in actual engineering. The embodiment of the present invention uses the correlation between different viewpoints and the division structure of the encoded viewpoints to set a depth threshold for the unencoded viewpoints to skip some unnecessary divisions, thereby reducing the encoding complexity. By analyzing the offset vectors and similarities of different viewpoint frames, as well as statistical information, a fast block division algorithm is assisted to minimize quality loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flowchart of the steps of a fast block partitioning method for multi-view video coding according to an embodiment of the present invention;

[0038] Figure 2 A depth map stored in a specific application instance;

[0039] Figure 3 Schematic diagram of the corresponding block to be encoded and DR in a specific application example. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0041] See also Figure 1 , which is a flowchart of a fast block partitioning method for multi-view video coding according to an embodiment of the present invention, includes the following steps:

[0042] S1, based on the pre-analysis lookahead process, reduces the resolution of the coded frame in the sequence to be coded to 1 / 2 of the original, and performs coarse intra-frame prediction and coarse inter-frame prediction in units of 16*16 pixels, and stores the costs obtained from the two predictions in the data structure of the corresponding frame; this step can quickly obtain some intermediate coded information through this simplified prediction mode, which can be used as auxiliary information for fine prediction.

[0043] CU stands for Coding Unit, and its size is not fixed. It is usually 8*8, 16*16, 32*32, 64*64, etc. Here, the size of 16*16 is fixed to simplify the pre-analysis process when performing coarse intra-frame prediction and coarse inter-frame prediction.

[0044] S2, take all viewpoint images of the first ten frames of the video sequence, perform four-fold downsampling, use the SIFT feature extraction algorithm to calculate the average offset vector V between the reference viewpoint and the referenced viewpoint offset (x, y); The effect of this step is to obtain the approximate offset between the two viewpoints to assist in establishing the connection between the reference and referenced blocks during the actual implementation of the algorithm, thereby preventing large quality losses.

[0045] S3, after the coding of the reference viewpoint is completed, the depth map of the reference viewpoint is stored according to the division structure of each coding tree unit CTU of the reference viewpoint. The storage unit of the depth map is the smallest CU; for example, a 32×32 CU with a depth of 1 is stored in the depth map in the form of 4×4 blocks with a depth of 1, such as Figure 2 As shown in the figure, except for the 32×32 CU in the upper left corner, the other three CUs display an optimal partition depth of 1 after encoding, so a value of 1 is assigned to their 32×32 partition size. The CU in the upper left corner displays further partition depths of 2 and 3 after encoding, so the corresponding 8×8 and 16×16 CUs are assigned values ​​of 3 and 2. This depth map can be used to determine the depth threshold of the entire CTU.

[0046] S4, for each CTU of the reference viewpoint, before actual encoding, according to its position (x, y) in the picture and the obtained average offset vector V offset (x, y), constructs the depth region DR based on the current block to be encoded; the construction of this region helps to determine the area range to be calculated when determining the depth threshold with reference to the coding block, so as to prevent covering too large an area and causing inaccurate depth threshold.

[0047] S5, placing the constructed DR in the corresponding position of the reference viewpoint, summarizing the depth distribution of the depth map in the DR, and taking the depth with the highest proportion as the maximum depth of the block to be coded of the reference viewpoint; the block to be coded takes the minimum value of the obtained coarse intra-frame prediction and coarse inter-frame prediction costs as the optimal cost, and adds the optimal cost of other coding blocks with the same depth, and compares the result with the optimal cost of its parent block. If the optimal cost of the parent block is smaller, the original maximum depth remains unchanged; if the optimal cost of the parent block is larger, the original maximum depth is increased by 1; the parent block is the block corresponding to the depth of the current coding block minus 1, and the block to be coded of the reference viewpoint will limit the number of times it is divided according to the maximum depth;

[0048] Right now:

[0049] Cost 最优 =min(Cost 粗帧内 , Cost 粗帧间 )

[0050]

[0051] Where t is the number of coding blocks at the same depth;

[0052] S6, refer to the viewpoint coding block for coding division. If the maximum division depth of the current coding block is greater than 2, the division depth distribution of the adjacent coded CTU of the current coding block is obtained; if the proportion of coding blocks with a depth less than 2 is greater than 50%, the depth threshold of the current coding block is set to 1, that is, the coding of depths 2 and 3 is skipped;

[0053] S7 , repeating the above steps S4 to S6 for all coding blocks of reference viewpoints until all coding blocks of the current viewpoint are encoded.

[0054] In specific application examples, coarse intra-frame prediction refers to calculating the SATD (Sum of Absolute Transformed Difference) of only a small number of prediction modes, including DC, PLANAR, ANGLE5, 10, 15, 20, 25 and 30, a total of 8 modes. Then, the mode with the best SATD is selected to perform SATD calculations for two prediction angles with a step size of 2. Finally, the best SATD is selected as the cost and its corresponding prediction mode. Coarse inter-frame prediction refers to performing motion search in the reduced reference frame to find the best matching block and calculating the SATD value of the residual as the cost of coarse inter-frame prediction. The mode with the best SATD is: comparing the SATD values ​​of all the above modes, the mode corresponding to the smallest SATD value is the best mode.

[0055] In another embodiment of the present invention, a fast block partitioning method for multi-view video coding is provided, wherein in S2, an average offset vector V between a reference viewpoint and a referenced viewpoint is calculated. offset (x,y) specifically includes:

[0056] Extract feature points, and assume that the feature point set of the two viewpoint images of the kth image is {P1, P2, ... P N} and {Q1,Q2,…Q N Then, the feature points are matched using the matching algorithm to obtain several matching pairs represented as (P i ,Q j ), for each matched feature point, calculate their coordinate offsets Δx and Δy:

[0057] Δx i =Q jx -P ix

[0058] Δy i =Q jy -P iy

[0059] Among them, P ix and P iy is the coordinate of the i-th feature point of the first viewpoint of the image, Q jx and Q jy is the coordinate of the jth feature point of the second viewpoint of the image;

[0060] Then calculate the average offset of all feature points:

[0061]

[0062] Where N is the number of matching pairs. After the average offset of the viewpoints corresponding to all images is calculated, the total average offset can be obtained. Since the above steps are performed on the image under four times downsampling, in order to correspond to the original image, the result of the total average offset needs to be multiplied by 4 to obtain the final V offset (x,y), that is: in:

[0063]

[0064]

[0065] When the vector component value is negative, it means it is shifting to the left or downward; when it is positive, it means it is shifting to the right or upward. offset The vector value of (x, y) needs to be aligned to the size of the CU, taking an integer multiple of the minimum CU size. Let the vector before alignment be a(x, y):

[0066] V offset (x,y)=a(x,y)+MiniSize CU -a(x,y)%MiniSize CU

[0067] Among them, MiniSize CU Represents the minimum value of the CU height set by the encoder, usually 8.

[0068] In another embodiment of the present invention, a fast block partitioning method for multi-view video coding is provided. In S4, before actual coding, the position (x, y) in the picture and the average offset vector V are obtained. offset (x, y), the depth region DR is constructed based on the current block to be coded, specifically including:

[0069] The height H of this area is: H = H CU +|V offsety |

[0070] The width W is: W = W CU +|V offsetx |

[0071] The starting point of the depth area is:

[0072] DR x =x+V offsetx

[0073] DR y =y+V offsety ;

[0074] Among them, H CU and W CUare the height and width of the CU to be encoded in pixels; V offsetx and V offsety The horizontal and vertical components of the average offset vector are used to construct the DR according to the starting point of the region and the H and W values. In a specific application example, Figure 3 , the dotted area is the corresponding block to be encoded, and the solid area is DR.

[0075] Table 1 shows the objective quality and encoding speed changes achieved using the method according to the present invention compared to the original encoder, using two 1080p and two 720p 3D sequences. As can be seen, the method according to the present invention reduces encoding complexity by approximately 30% on average, with a performance loss of approximately 1%. It also achieves a significantly better acceleration effect for the higher-resolution 1080p sequence.

[0076] Table 1 Performance comparison between the method of the embodiment of the present invention and the existing method

[0077]

[0078] The fast block partitioning method for multi-view video coding described above uses the depth distribution of the coded blocks of the reference view to set a threshold for the depth of the coded blocks of the reference view under the same Point of View (POC) to skip unnecessary partitions. By analyzing the offset vectors and similarities of frames from different viewpoints, as well as statistical information, the fast block partitioning algorithm is assisted to minimize quality loss.

[0079] It should be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the present invention have been described in conjunction with the accompanying drawings, it should be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A fast block partitioning method for multi-view video coding, characterized in that: The following steps are involved: S1, based on the pre-analysis lookahead process, reduces the resolution of the coded frame in the sequence to be coded to 1 / 2 of the original, and performs coarse intra-frame prediction and coarse inter-frame prediction in units of 16*16 pixels, and stores the costs obtained from the two predictions in the data structure of the corresponding frame; S2, take all viewpoint images of the first ten frames of the video sequence, perform four-fold downsampling, use the SIFT feature extraction algorithm to calculate the average offset vector V between the reference viewpoint and the referenced viewpoint offset (x,y); S3, after the encoding of the reference viewpoint is completed, storing the depth map of the reference viewpoint according to the division structure of each coding tree unit CTU of the reference viewpoint, and the storage unit of the depth map is the minimum CU; S4, for each CTU of the reference viewpoint, before actual encoding, according to its position (x, y) in the picture and the obtained average offset vector V offset (x, y), build the depth region DR based on the current block to be encoded; S5, placing the constructed DR at the corresponding position of the reference viewpoint, summarizing the depth distribution of the depth map in the DR, and taking the depth with the highest proportion as the maximum depth of the block to be encoded of the reference viewpoint; The minimum value of the coarse intra prediction and coarse inter prediction costs is taken as the optimal cost for the block to be coded, and the optimal cost of other coding blocks at the same depth is added. The result is compared with the optimal cost of its parent block. If the optimal cost of the parent block is smaller, the original maximum depth remains unchanged. If the optimal cost of the parent block is larger, the original maximum depth is increased by 1. The parent block is the block corresponding to the depth of the current coding block minus 1. The block to be coded at the reference viewpoint will limit the number of times it is divided according to the maximum depth. S6, refer to the viewpoint coding block for coding division. If the maximum division depth of the current coding block is greater than 2, the division depth distribution of the adjacent coded CTU of the current coding block is obtained; if the proportion of coding blocks with a depth less than 2 is greater than 50%, the depth threshold of the current coding block is set to 1, that is, the coding of depths 2 and 3 is skipped; S7 , repeating the above steps S4 to S6 for all coding blocks of reference viewpoints until all coding blocks of the current viewpoint are encoded.

2. The fast block partitioning method for multi-view video coding according to claim 1, wherein: The coarse intra prediction refers to the calculation of the SATD (Sum of Absolute The algorithm uses 8 kinds of Transformed Difference (SATD), including DC, PLANAR, ANGLE5, 10, 15, 20, 25 and 30. Then, the SATD mode with the best SATD is selected to calculate the SATD of two prediction angles with a step size of 2. Finally, the best SATD is selected as the cost and its corresponding prediction mode.

3. The fast block partitioning method for multi-view video coding according to claim 1, wherein: The coarse inter-frame prediction refers to performing a motion search in the reduced reference frame to find the best matching block and calculating the SATD value of the residual as the cost of the coarse inter-frame prediction.

4. The fast block partitioning method for multi-view video coding according to any one of claims 1 to 3, wherein: The average offset vector V between the reference viewpoint and the referenced viewpoint is calculated offset (x,y) specifically includes: Extract feature points, and assume that the feature point set of the two viewpoint images of the kth image is {P1, P2, ... P N } and {Q1,Q2,…Q N Then, the feature points are matched using the matching algorithm to obtain several matching pairs represented as (P i ,Q j ), for each matched feature point, calculate their coordinate offsets Δx and Δy: Δx i =Q jx -P ix Δy i =Q jy -P iy Among them, P ix and P iy is the coordinate of the i-th feature point of the first viewpoint of the image, Q jx and Q jy is the coordinate of the jth feature point of the second viewpoint of the image; Then calculate the average offset of all feature points: Where N is the number of matching pairs; after the average offset of the viewpoints corresponding to all images is calculated, the total average offset can be obtained; since the above steps are performed on the image under four times downsampling, in order to correspond to the original image, the result of the total average offset needs to be multiplied by 4 to obtain the final V offset (x,y), that is: in: When the vector component value is negative, it means it is shifting to the left or downward; when it is positive, it means it is shifting to the right or upward. offset The vector value of (x, y) needs to be aligned to the size of the CU, taking an integer multiple of the minimum CU size. Let the vector before alignment be a(x, y): V offset (x,y)=a(x,y)+MiniSize CU -a(x,y)%MiniSize CU Among them, MiniSize CU Represents the minimum value of the CU height set by the encoder, usually 8.

5. The fast block partitioning method for multi-view video coding according to claim 4, wherein: The actual encoding is based on its position (x, y) in the picture and the average offset vector V obtained offset (x, y), the depth region DR is constructed based on the current block to be coded, specifically including: The height H of this area is: H = H CU +|V offsety | The width W is: W = W CU +|V offsetx | The starting point of the depth area is: DR x =x+V offsetx DR y =y+V offsety ; Among them, H CU and W CU are the height and width of the CU to be encoded in pixels; V offsetx and V offsety The horizontal and vertical component values ​​of the average offset vector are used. According to the starting point of the area and the H and W values, the DR can be constructed.

Images