Fast block partitioning method for multi-view video coding

Abstract

The invention discloses a fast block division method for multi-view video coding, which comprises the following steps of: S1, reducing the resolution ratio of a coding frame in a sequence to be coded to be 1 / 2 of the original resolution ratio, and performing coarse intra-frame prediction and coarse inter-frame prediction by taking 16 * 16 pixels as units; s2, calculating an average offset vector Voffset (x, y) between the reference viewpoint and the referenced viewpoint by using an SIFT (Scale Invariant Feature Transform) feature extraction algorithm; s3, after the coding of the referenced viewpoint is completed, storing a depth map of the referenced viewpoint according to the division structure of each coding tree unit CTU of the referenced viewpoint; s4, for each CTU of the reference viewpoint, according to the position (x, y) of the CTU in the picture and the obtained average offset vector Voffset (x, y) before actual coding, constructing a depth region DR by taking the current to-be-coded block as a benchmark; s5, placing the constructed DR in the corresponding position of the referenced viewpoint, and summarizing the depth distribution of the depth map in the DR; s6, carrying out coding division on the reference viewpoint coding block; and S7, repeating the steps S4 to S6 on the coding blocks of all the reference viewpoints until all the coding blocks of the current viewpoint are coded.

Description

Technical Field

[0001] The invention belongs to the technical field of video coding, and in particular relates to a fast block division method for multi-viewpoint video coding. Background Art

[0002] In recent years, with the rapid development of multimedia technology, people are no longer satisfied with simple high-definition video applications, but hope to receive and experience video content in a multi-faceted manner through their own senses. This demand has promoted the development of ordinary single-view video content applications to multi-view video applications. Multi-view video refers to video that is recorded by multiple cameras at different angles at the same time. In this type of video application, multi-view video compression is an important research and component. The international standards group has expanded on the basis of the single-view compression standard and released standards for multi-view video. The most advanced one is the MV-HEVC standard based on the HEVC standard extension. Compared with multi-channel simulcast encoding, the technology used in the MV-HEVC standard can reduce the bit rate by 20% to 30% under the same picture quality.

[0003] The prediction technology used by MV-HEVC in the process of predictive coding is basically the same as that of HEVC, including 35 angle predictions for intra-frame prediction; AMVP, Merge, and Skip modes for inter-frame prediction. Similarly, in the process of rate-distortion optimization, CTU (Coding Tree Unit) needs to be recursively divided according to 64×64, 32×32, 16×16, and 8×8. However, the increase in viewpoints to be compressed also brings a more complicated encoding process. In addition to the intra-frame prediction and inter-frame prediction of the newly added viewpoints themselves, different viewpoints in the same POC (Picture Order Count) will use a new prediction coding method-inter-view prediction to pursue smaller residuals. The principle of inter-view prediction is basically the same as that of inter-frame prediction, and the main difference lies in the different reference frames. The reference frame of inter-frame prediction is an adjacent frame in the time domain, and the motion estimation performed on this frame is called ME (Motion Estimation). The reference frame for inter-view prediction is the adjacent viewpoint frame at the same time, and the motion estimation performed on this frame is called disparity estimation DE (Disparity Estimation). The newly added mode also requires the recursive block partitioning process to execute more prediction modes to pursue the best encoding quality, but this makes the complexity of multi-view encoding much higher than using conventional coding standards (such as HEVC) to encode different viewpoint sequences separately.

[0004] According to the actual MV-HEVC code stream, since the reference viewpoint and the referenced viewpoint usually have a high degree of similarity, the reference viewpoint is the viewpoint that refers to the referenced viewpoint during inter-view prediction to reduce the prediction residual. In binocular video, it is usually the auxiliary viewpoint. The referenced viewpoint is used to provide a reference for inter-view prediction for the reference viewpoint. In binocular video, it is usually the main viewpoint. In the division structure of the reference viewpoint, there are rarely coding blocks with higher depths. The proportion of coding blocks that are not divided is as high as 76%, and the proportion of coding blocks that are only divided once is also 17%. The remaining only 7% are coding blocks with more detailed divisions. Without any fast algorithm optimization, these coding blocks that do not need to be divided or only need to be divided once still need to perform a complex rate-distortion optimization process, try multiple prediction modes, and finally get the optimal result. Taking a 64×64 CU as an example, if it is assumed that each recursive division is simplified to perform intra-frame prediction, inter-frame prediction, and inter-view prediction, the CU will execute a total of 3×(1+4+16+64)=255 prediction modes. If the optimal mode of the CU is not to be divided, the execution of the remaining 255-3=252 prediction modes is redundant, wasting a lot of time. Summary of the invention

[0005] In view of the above problems, the present invention provides a fast block partitioning method for multi-viewpoint video coding, which is used to utilize the correlation between different viewpoints and the partitioning structure of encoded viewpoints, set depth thresholds for unencoded viewpoints, and skip some unnecessary partitions, thereby reducing coding complexity.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A fast block partitioning method for multi-view video coding comprises the following steps:

[0008] S1, based on the pre-analysis lookahead process, the resolution of the coded frame in the sequence to be coded is reduced to 1 / 2 of the original, and coarse intra-frame prediction and coarse inter-frame prediction are performed in units of 16*16 pixels, and the costs obtained by the two predictions are stored in the data structure of the corresponding frame;

[0009] S2, take all viewpoint images of the first ten frames of the video sequence, perform four-fold downsampling, use the SIFT feature extraction algorithm to calculate the average offset vector V between the reference viewpoint and the referenced viewpoint offset (x,y);

[0010] S3, after the coding of the referenced viewpoint is completed, according to the division structure of each coding tree unit CTU of the referenced viewpoint, the depth map of the referenced viewpoint is stored, and the storage unit of the depth map is the minimum CU;

[0011] S4, for each CTU of the reference viewpoint, before actual encoding, according to its position (x, y) in the picture and the obtained average offset vector V offset (x, y), build the depth region DR based on the current block to be encoded;

[0012] S5, placing the constructed DR in the corresponding position of the reference viewpoint, summarizing the depth distribution of the depth map in the DR, and taking the depth with the highest proportion as the maximum depth of the block to be encoded of the reference viewpoint; the block to be encoded takes the minimum value of the obtained coarse intra-frame prediction and coarse inter-frame prediction costs as the optimal cost, and adds the optimal cost of other coding blocks with the same depth, and compares the result with the optimal cost of its parent block. If the optimal cost of the parent block is smaller, the original maximum depth remains unchanged. If the optimal cost of the parent block is larger, the original maximum depth is increased by 1; the parent block is the block corresponding to the depth of the current coding block minus 1, and the block to be encoded of the reference viewpoint will limit the number of divisions according to the maximum depth;

[0013] S6, refer to the viewpoint coding block for coding division. If the maximum division depth of the current coding block is greater than 2, the division depth distribution of the adjacent coded CTU of the current coding block is obtained; if the proportion of coding blocks with a depth less than 2 is greater than 50%, the depth threshold of the current coding block is set to 1, that is, the coding of depths 2 and 3 is skipped;

[0014] S7, repeating the above steps S4 to S6 for all coding blocks of reference viewpoints until all coding blocks of the current viewpoint are encoded.

[0015] In a possible implementation manner, the coarse intra-frame prediction refers to calculating only the SATD (Sum of Absolute Transformed Difference) of a small number of prediction modes, including 8 modes, namely DC, PLANAR, ANGLE5, 10, 15, 20, 25 and 30, and then selecting the mode with the best SATD to perform SATD calculations for two prediction angles with a step size of 2, and finally selecting the best SATD as the cost and its corresponding prediction mode.

[0016] In a possible implementation manner, the coarse inter-frame prediction refers to performing a motion search in the reduced reference frame to find an optimal matching block, and calculating a SATD value of the residual as the cost of the coarse inter-frame prediction.

[0017] In a possible implementation manner, the calculation of the average offset vector V between the reference viewpoint and the referenced viewpoint offset (x,y) specifically includes:

[0018] Extract feature points. Let the feature point set of the two viewpoint images of the kth image be {P1, P2, ... P N} and {Q1,Q2,…Q N}; Then, the feature points are matched using the matching algorithm to obtain several matching pairs represented as (P i ,Q j ), for each matched feature point, calculate their coordinate offsets Δx and Δy:

[0019] Δx i =Q "x -P ix

[0020] Δy i =Q "$ -P i$

[0021] Among them, P ix and P i$ is the coordinate of the i-th feature point of the first viewpoint of the image, Q "x and Q "$ is the coordinate of the jth feature point of the second viewpoint of the image;

[0022] Then calculate the average offset of all feature points:

[0023]

[0024] Where N is the number of matching pairs. After the average offset of the viewpoints corresponding to all images is calculated, the total average offset can be obtained. Since the above steps are performed on the image under four times downsampling, in order to correspond to the original image, the result of the total average offset needs to be multiplied by 4 to obtain the final V offset (x,y), that is: in:

[0025]

[0026] When the vector component value is negative, it means it is shifted to the left or downward; when it is positive, it means it is shifted to the right or upward. offset The vector value of (x, y) needs to be aligned to the size of the CU, taking an integer multiple of the minimum CU size. Let the vector before alignment be a(x, y):

[0027] V offset (x,y)=a(x,y)+MiniSize CU -a(x,y)%MiniSize CU

[0028] Among them, MiniSize CU Represents the minimum value of the CU height set by the encoder, usually 8.

[0029] In a possible implementation manner, before actual encoding, the position (x, y) in the picture and the average offset vector V obtained are obtained. offset (x, y), the depth region DR is constructed based on the current block to be encoded, including:

[0030] The height H of this area is: H = H CU +|V offsety |

[0031] The width W is: W = W CU +|V offsetx |

[0032] The starting point of the depth area is:

[0033] DR x =x+V offsetx

[0034] DR y =y+V offsety ;

[0035] Among them, H CU and W CU are the height and width of the CU to be encoded, respectively; V offsetx and V offsety The horizontal and vertical components of the average offset vector are used to construct the DR according to the starting point of the area and the H and W values.

[0036] The use of the present invention has the following beneficial effects: the existing multi-view video compression standards, such as MV-HEVC, have added new viewpoints, resulting in more prediction modes to be executed during the encoding process, which makes the rate-distortion optimization process take more time, thereby increasing the complexity of the overall encoding, making the compression of multi-view video difficult to apply in actual engineering. The embodiment of the present invention uses the correlation between different viewpoints and the division structure of the encoded viewpoints to set a depth threshold for the unencoded viewpoints to skip some unnecessary divisions, thereby reducing the encoding complexity. By analyzing the offset vectors and similarities of different viewpoint frames, as well as statistical information, the fast block division algorithm is assisted to minimize quality loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flowchart of the steps of a fast block partitioning method for multi-view video coding according to an embodiment of the present invention;

[0038] Figure 2 A depth map stored in a specific application instance;

[0039] Figure 3 It is a schematic diagram of the corresponding block to be encoded and DR in a specific application example. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] See also Figure 1 , which is a flowchart of a fast block partitioning method for multi-view video coding according to an embodiment of the present invention, comprises the following steps:

[0042] S1, based on the pre-analysis lookahead process, reduces the resolution of the coded frame in the sequence to be coded to 1 / 2 of the original, and performs coarse intra-frame prediction and coarse inter-frame prediction in units of 16*16 pixels, and stores the costs obtained from the two predictions in the data structure of the corresponding frame; this step can quickly obtain some intermediate information of the code through this simplified prediction mode, which can be used as auxiliary information for fine prediction.

[0043] CU stands for Coding Unit, and its size is not fixed, usually 8*8, 16*16, 32*32, 64*64, etc. Here, the size of 16*16 is fixed to simplify the pre-analysis process when performing coarse intra-frame prediction and coarse inter-frame prediction.

[0044] S2, take all viewpoint images of the first ten frames of the video sequence, perform four-fold downsampling, use the SIFT feature extraction algorithm to calculate the average offset vector V between the reference viewpoint and the referenced viewpoint offset (x, y); The effect of this step is to obtain the approximate offset between the two viewpoints to assist in the actual implementation of the algorithm, to establish the connection between the reference and referenced blocks, and to prevent a large quality loss.

[0045] S3, after the coding of the referenced viewpoint is completed, the depth map of the referenced viewpoint is stored according to the division structure of each coding tree unit CTU of the referenced viewpoint. The storage unit of the depth map is the smallest CU; for example, a 32×32 CU with a depth of 1 is stored in the depth map in the form of 4×4 blocks with a depth of 1, such as Figure 2 As shown in the figure. In addition to the 32×32 CU in the upper left corner, the other three CUs show the optimal division depth of 1 after encoding, so the value of 1 is assigned within the 32×32 range of their division size. The CU in the upper left corner shows further division depths of 2 and 3 after encoding, so the values ​​of 3 and 2 are assigned to the 8×8 and 16×16 CUs corresponding to their division depths. This depth map can be used to determine the depth threshold of the entire CTU.

[0046] S4, for each CTU of the reference viewpoint, before actual encoding, according to its position (x, y) in the picture and the obtained average offset vector V offset (x, y), construct the depth region DR based on the current block to be encoded; the construction of this region helps to determine the area range to be calculated when the depth threshold is determined with reference to the encoding block, so as to prevent covering too large an area and causing inaccurate depth threshold.

[0047] S5, placing the constructed DR in the corresponding position of the reference viewpoint, summarizing the depth distribution of the depth map in the DR, and taking the depth with the highest proportion as the maximum depth of the block to be encoded of the reference viewpoint; the block to be encoded takes the minimum value of the obtained coarse intra-frame prediction and coarse inter-frame prediction costs as the optimal cost, and adds the optimal cost of other coding blocks with the same depth, and compares the result with the optimal cost of its parent block. If the optimal cost of the parent block is smaller, the original maximum depth remains unchanged. If the optimal cost of the parent block is larger, the original maximum depth is increased by 1; the parent block is the block corresponding to the depth of the current coding block minus 1, and the block to be encoded of the reference viewpoint will limit the number of divisions according to the maximum depth;

[0048] Right now:

[0049] Cost 最优 =min(Cost 粗帧内 , Cost 粗帧间 )

[0050]

[0051] Where t is the number of coding blocks at the same depth;

[0052] S6, refer to the viewpoint coding block for coding division. If the maximum division depth of the current coding block is greater than 2, the division depth distribution of the adjacent coded CTU of the current coding block is obtained; if the proportion of coding blocks with a depth less than 2 is greater than 50%, the depth threshold of the current coding block is set to 1, that is, the coding of depths 2 and 3 is skipped;

[0053] S7, repeating the above steps S4 to S6 for all coding blocks of reference viewpoints until all coding blocks of the current viewpoint are encoded.

[0054] In specific application examples, coarse intra-frame prediction refers to calculating the SATD (Sum of Absolute Transformed Difference) of only a small number of prediction modes, including DC, PLANAR, ANGLE5, 10, 15, 20, 25 and 30, a total of 8 types. Then, the mode with the best SATD is selected to perform SATD calculations for two prediction angles with a step size of 2, and finally the best SATD is selected as the cost and its corresponding prediction mode. Coarse inter-frame prediction refers to performing motion search in the reduced reference frame to find the best matching block, and calculating the SATD value of the residual as the cost of coarse inter-frame prediction. The best SATD mode is: compare the SATD values ​​of all the above modes, and the mode corresponding to the smallest SATD value is the best mode.

[0055] In a fast block partitioning method for multi-view video coding according to another embodiment of the present invention, in S2, an average offset vector V between a reference viewpoint and a referenced viewpoint is calculated. offset (x,y) specifically includes:

[0056] Extract feature points. Let the feature point set of the two viewpoint images of the kth image be {P1, P2, ... P N} and {Q1,Q2,…Q N}; Then, the feature points are matched using the matching algorithm to obtain several matching pairs represented as (P i ,Q j ), for each matched feature point, calculate their coordinate offsets Δx and Δy:

[0057] Δx i =Q "x -P ix

[0058] Δy i =Q "$ -P i$

[0059] Among them, P ix and P i$ is the coordinate of the i-th feature point of the first viewpoint of the image, Q "x and Q "$ is the coordinate of the jth feature point of the second viewpoint of the image;

[0060] Then calculate the average offset of all feature points:

[0061]

[0062] Where N is the number of matching pairs. After the average offset of the viewpoints corresponding to all images is calculated, the total average offset can be obtained. Since the above steps are performed on the image under four times downsampling, in order to correspond to the original image, the result of the total average offset needs to be multiplied by 4 to obtain the final V offset (x,y), that is: in:

[0063]

[0064]

[0065] When the vector component value is negative, it means it is shifted to the left or downward; when it is positive, it means it is shifted to the right or upward. offset The vector value of (x, y) needs to be aligned to the size of the CU, taking an integer multiple of the minimum CU size. Let the vector before alignment be a(x, y):

[0066] V offset (x,y)=a(x,y)+MiniSize CU -a(x,y)%MiniSize CU

[0067] Among them, MiniSize CU Represents the minimum value of the CU height set by the encoder, usually 8.

[0068] In another embodiment of the present invention, a fast block partitioning method for multi-view video coding is provided. In S4, before actual coding, the position (x, y) in the picture and the average offset vector V are obtained. offset (x, y), the depth region DR is constructed based on the current block to be encoded, including:

[0069] The height H of this area is: H = H CU +|V offsety |

[0070] The width W is: W = W CU +|V offsetx |

[0071] The starting point of the depth area is:

[0072] DR x =x+V offsetx

[0073] DR y =y+V offsety ;

[0074] Among them, H CU and W CUare the height and width of the CU to be encoded, respectively; V offsetx and V offsety is the horizontal and vertical component values ​​of the average offset vector. According to the starting point of the area and the H and W values, the DR can be constructed. In a specific application example, Figure 3 , the dotted area is the corresponding block to be encoded, and the solid area is DR.

[0075] Table 1 shows the objective quality change and encoding speed change of the method of the embodiment of the present invention compared with the original encoder under the test of two 1080p and two 720p 3D sequences. It can be seen that the method of the embodiment of the present invention can reduce the encoding complexity by about 30% on average, and the performance loss is about 1%. And for the 1080p sequence with higher resolution, it has a better acceleration effect.

[0076] Table 1 Performance comparison between the method of the present invention and the existing method

[0077]

[0078] The fast block partitioning method for multi-view video coding set above uses the depth distribution of the coded block partitioning of the reference viewpoint to set a threshold for the partitioning depth of the coded block of the reference viewpoint under the same POC to skip unnecessary partitioning. By analyzing the offset vectors and similarities of different viewpoint frames, as well as statistical information, the fast block partitioning algorithm is assisted to minimize quality loss.

[0079] It should be understood that the exemplary embodiments described herein are illustrative rather than restrictive. Although one or more embodiments of the present invention have been described in conjunction with the accompanying drawings, it should be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A fast block partitioning method for multi-view video coding, characterized in that: The following steps are involved: S1, based on the pre-analysis lookahead process, the resolution of the coded frame in the sequence to be coded is reduced to 1 / 2 of the original, and coarse intra-frame prediction and coarse inter-frame prediction are performed in units of 16*16 pixels, and the costs obtained by the two predictions are stored in the data structure of the corresponding frame; S2, take all viewpoint images of the first ten frames of the video sequence, perform four-fold downsampling, use the SIFT feature extraction algorithm to calculate the average offset vector V between the reference viewpoint and the referenced viewpoint offset (x,y); S3, after the coding of the referenced viewpoint is completed, according to the division structure of each coding tree unit CTU of the referenced viewpoint, the depth map of the referenced viewpoint is stored, and the storage unit of the depth map is the minimum CU; S4, for each CTU of the reference viewpoint, before actual encoding, according to its position (x, y) in the picture and the obtained average offset vector V offset (x, y), build the depth region DR based on the current block to be encoded; S5, placing the constructed DR in the corresponding position of the reference viewpoint, summarizing the depth distribution of the depth map in the DR, and taking the depth with the highest proportion as the maximum depth of the block to be encoded of the reference viewpoint; The block to be coded takes the minimum value of the obtained coarse intra-frame prediction and coarse inter-frame prediction costs as the optimal cost, and adds the optimal cost of other coding blocks with the same depth, and compares the result with the optimal cost of its parent block. If the optimal cost of the parent block is smaller, the original maximum depth remains unchanged. If the optimal cost of the parent block is larger, the original maximum depth is increased by 1; the parent block is the block corresponding to the depth of the current coding block minus 1, and the block to be coded at the reference viewpoint will limit the number of divisions according to the maximum depth; S6, refer to the viewpoint coding block for coding division. If the maximum division depth of the current coding block is greater than 2, the division depth distribution of the adjacent coded CTU of the current coding block is obtained; if the proportion of coding blocks with a depth less than 2 is greater than 50%, the depth threshold of the current coding block is set to 1, that is, the coding of depths 2 and 3 is skipped; S7, repeating the above steps S4 to S6 for all coding blocks of reference viewpoints until all coding blocks of the current viewpoint are encoded.

2. The fast block partitioning method for multi-view video coding according to claim 1, characterized in that: The coarse intra prediction refers to the calculation of the SATD (Sum of Absolute Then, the SATD mode with the best SATD is selected to calculate the SATD of two prediction angles with a step size of 2. Finally, the best SATD is selected as the cost and its corresponding prediction mode.

3. The fast block partitioning method for multi-view video coding according to claim 1, characterized in that: The coarse inter-frame prediction refers to performing motion search in the reduced reference frame to find the best matching block and calculating the SATD value of the residual as the cost of the coarse inter-frame prediction.

4. The fast block partitioning method for multi-view video coding according to any one of claims 1 to 3, characterized in that: The calculation of the average offset vector V between the reference viewpoint and the referenced viewpoint offset (x,y) specifically includes: Extract feature points. Let the feature point set of the two viewpoint images of the kth image be {P1, P2, ... P N } and {Q1,Q2,…Q N }; Then, the feature points are matched using the matching algorithm to obtain several matching pairs represented as (P i ,Q j ), for each matched feature point, calculate their coordinate offsets Δx and Δy: Δx i =Q jx -P ix Δy i =Q jy -P iy Among them, P ix and P iy is the coordinate of the i-th feature point of the first viewpoint of the image, Q jx and Q jy is the coordinate of the jth feature point of the second viewpoint of the image; Then calculate the average offset of all feature points: Where N is the number of matching pairs; after the average offset of the viewpoints corresponding to all images is calculated, the total average offset can be obtained; since the above steps are performed on the image under four times downsampling, in order to correspond to the original image, the result of the total average offset needs to be multiplied by 4 to obtain the final V offset (x,y), that is: in: When the vector component value is negative, it means it is shifted to the left or downward; when it is positive, it means it is shifted to the right or upward. offset The vector value of (x, y) needs to be aligned to the size of the CU, taking an integer multiple of the minimum CU size. Let the vector before alignment be a(x, y): V offset (x,y)=a(x,y)+MiniSize CU -a(x,y)%MiniSize CU Among them, MiniSize CU Represents the minimum value of the CU height set by the encoder, usually 8.

5. The fast block partitioning method for multi-view video coding as claimed in claim 4, characterized in that: The actual encoding is based on its position (x, y) in the picture and the average offset vector V obtained offset (x, y), the depth region DR is constructed based on the current block to be encoded, including: The height H of this area is: H = H CU +|V offsety | The width W is: W = W CU +|V offsetx | The starting point of the depth area is: DR x =x+V offsetx DR y =y+V offsety ; Among them, H CU and W CU are the height and width of the CU to be encoded, respectively; V offsetx and V offsety The horizontal and vertical components of the average offset vector are used to construct the DR according to the starting point of the area and the H and W values.

Images