Cutree technology optimization method for multi-view video coding

Abstract

The invention discloses a Cutree technology optimization method for multi-view video coding, and the method comprises the following steps: S1, based on a pre-analysis lookahead process, selecting to-be-coded frames of all viewpoints of a GOP number to enter a lookahead sequence, reducing the resolution of the coded frames to 1 / 2 of the original resolution for storage, dividing coding blocks by taking 16 * 16 pixels as units, and storing the coding blocks in the lookahead sequence; performing coarse intra-frame prediction, coarse inter-frame prediction and coarse inter-viewpoint prediction; s2, two non-B frames form a mini GOP, the number of viewpoints is set to be N, and N mini GOPs exist at the moment; s3, for the mini GOP of the two viewpoints, setting the calculation sequence of the frames as b-auxiliary 1b-main 1b-auxiliary 2b-main 2B-auxiliary B-main P-auxiliary 2P-main 2P-auxiliary 1P-main 1; s4, calculating the propagation information sum propalatee of the current block propagated to the current block from the other blocks referring to the current block; s5, after the current block is completely propagated, entering the next block to continue propagation; and S6, if the propalatecost of all the blocks of a certain frame is calculated, adjusting the QP of each block according to the value, so as to allocate different code rates for different blocks according to different importance.

Description

Technical Field

[0001] The present invention belongs to the technical field of video coding, and in particular relates to a Cutree technology optimization method for multi-view video coding. Background Art

[0002] In recent years, with the continuous improvement of video processing technology and imaging display technology, the demand for video content has steadily developed in the direction of high definition and stereoscopic. For high definition, the Video Coding Experts Group of the International Telecommunication Union Telecommunication Standardization Organization and the Moving Picture Experts Group of the International Organization for Standardization jointly launched standards such as AVC and HEVC for efficient compression of high-definition video. For stereoscopic, the standardization organization launched corresponding extended standards based on the above standards: MVC and MV-HEVC, which are used to compress multi-viewpoint video content based on binocular video. These video contents can be used as the representation form of most 3D videos to bring viewers a roaming experience from different perspectives.

[0003] However, these extended standards only make adaptive modifications to multi-view video compression at the high-level syntax elements and frame level, but do not make any underlying changes to some existing technologies for single-view. This causes certain problems when multi-view video standards directly reuse these technologies.

[0004] Cutree technology is a technology used by the rate control module under the HEVC standard to effectively improve the subjective and objective quality. Its principle is to adjust the QP according to the reference degree of the block to be encoded. offset , so as to reasonably allocate the bits of each coding block. If the block to be coded contributes more information to the subsequent frames, the importance of the block is considered to be higher, and the coding quality of the block needs to be improved, so QP should be set. offset Negative value to reduce QP. offset The reference degree described in this principle is obtained based on the pre-analysis (lookahead) process of encoding. If an I frame is referenced by a P frame and multiple B frames, the reference degree of the I frame is very high and will be assigned a lower QP. Non-reference B frames will not participate in the adjustment of Cutree because they have no contribution, that is, the Cutree technology only acts on frames that can be used as references.

[0005] Multi-view video refers to a video that is recorded by multiple cameras at different angles at the same time. In the framework of multi-view video coding, the reference relationship is more complex than that of single-channel video coding. In addition to inter-frame prediction, inter-view prediction is also added: under the same POC, the frame of the main viewpoint can be predicted by the auxiliary viewpoint. The increase in reference relationships causes certain errors in the application of the original Cutree technology. For example, when the cost of the inter-frame prediction and the inter-view prediction obtained in the lookahead process are very close, it is impossible to accurately judge which of the two modes is better. At this time, if only one of the modes is arbitrarily selected and the corresponding reference frame is propagated, the importance of the reference frame of the other mode may be calculated incorrectly, which will lead to errors in its bit rate allocation and affect the bit rate allocation of this video. Therefore, in view of this situation, weight allocation optimization should be performed. Summary of the invention

[0006] In view of the above problems, the present invention provides a method for optimizing the Cutree technology for multi-view video coding, which uses pre-analysis and related information between viewpoints to optimize the Cutree technology in a targeted manner to further improve the efficiency of multi-view video coding standard in compressing multi-view videos.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A method for optimizing the Cutree technology for multi-view video coding comprises the following steps:

[0009] S1, based on the pre-analysis lookahead process, select all the frames to be coded from the viewpoints of a GOP into the lookahead sequence, reduce the resolution of these coded frames to 1 / 2 of the original and store them, divide the coding blocks into 16*16 pixels, perform coarse intra-frame prediction, coarse inter-frame prediction and coarse inter-viewpoint prediction, and the three costs are recorded as C intra , C inter , and C inter-view , and stored in the data structure of the corresponding frame;

[0010] S2, enter the lookahead sequence again, traverse forward from the last frame, first find the last non-B frame, and then find all the viewpoint frames of the non-B frame. The sequence numbers of these frames in the sequence should be continuous; then search forward to find the second non-B frame and all its viewpoint frames. These two non-B frames form a miniGOP. Assume that the number of viewpoints is N, then there are N miniGOPs at this time;

[0011] S3, for the miniGOP of these two viewpoints, set the frame calculation order to b 辅1 b 主1 b 辅2 b主2 B 辅 B 主 P 辅2 P 主2 P 辅 1P 主1 ;

[0012] S4, according to the set calculation order, extract the corresponding C of each block in the current frame intra , C inter and C inter-view In Cutree, when calculating the importance, the block traversal is performed in units of 16*16 pixels, and then one of the 16*16 pixel blocks in the current frame is used to calculate the sum of the propagation information propagated to the current block from the other blocks that refer to it. cost , which depends on the amount of information propagated by the block referring to the current block, namely:

[0013]

[0014] Among them, propagate amount _ref i represents the amount of information propagated from the i-th reference current block to the current block; for the i-th reference block, the total amount of information to be propagated is:

[0015] Propagate amount =(C intra *invQScales+propagate cost )*Fpropagate

[0016] Among them, invQScales is a fixed value related to the frame-level QP of the frame where the current block is located, and Fpropagate is the ratio of information obtained from the reference frame;

[0017] S5, propagate according to the ratio of pixel area amount Split and pass them to different covered blocks, and further propagate amount The reference blocks are equally divided into two reference frames. After the current block is propagated, the next block is entered for further propagation.

[0018] S6, if the propagate of all blocks of a frame cost After the calculation is completed, the QP of each block is adjusted according to the value to assign different bit rates to different blocks according to their different importance; QP adjustment value QP offset The calculation formula is:

[0019]

[0020] Among them, strength is the adjustment strength set in the Cutree technology, which is generally set to 2; each block is based on the calculated QP offset Adjust QP to reasonably allocate bit rate and improve the subjective and objective quality of the video.

[0021] In one possible implementation, in S1, the coarse intra-frame prediction refers to calculating only a small number of SATDs of prediction modes compared to conventional intra-frame prediction, including 8 modes in total, namely DC, PLANAR, ANGLE5, 10, 15, 20, 25 and 30. Then, the mode with the best SATD is selected to perform SATD calculations at two prediction angles with a step size of 2. Finally, the best SATD is selected as the cost and its corresponding prediction mode is selected as the optimal prediction angle of the coarse intra-frame prediction mode.

[0022] In a possible implementation, in S1, the coarse inter-frame prediction refers to performing a motion search in the reduced reference frame to find the best matching block, and calculating a SATD value of the residual as the cost of the coarse inter-frame prediction.

[0023] In a possible implementation, in S1, the coarse inter-viewpoint prediction refers to performing motion search using a referenced viewpoint frame at the same POC as a reference frame to find the best matching block, and calculating a residual SATD value as the cost of the coarse inter-viewpoint prediction.

[0024] In a possible implementation, for the frame of the main viewpoint, since inter-view prediction is not performed, Fpropagate is only calculated by C intra , C inter Perform calculations;

[0025]

[0026] In a possible implementation, for frames from non-primary viewpoints, the influence of inter-view prediction needs to be further considered. To calculate Fpropagate, first calculate C inter With C inter-view The difference ratio ΔC (%):

[0027]

[0028] Based on the relationship between the difference ratio and the given threshold Sh, as well as the referenced main viewpoint frame type and the POC distance of the temporal reference, the value of Fpropagate is as follows:

[0029]

[0030] The threshold Sh is fixed at 0.4.

[0031] The present invention has the following beneficial effects:

[0032] (1) Based on the reference characteristics of multi-view video, the propagation order and calculation order in the implementation process of Cutree are reconstructed to conform to the actual encoding order of multi-view video encoding, making the bit rate allocation result more accurate.

[0033] (2) The encoder's pre-analysis information (including SATD, mode, etc. based on 16*16CU intra-frame prediction and inter-frame prediction) and the similarity information between viewpoints are used to optimize the back-propagation process of the Cutree technology in multi-view video coding and set the weight coefficient of the propagation information. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of the steps of the Cutree technology optimization method for multi-view video coding according to an embodiment of the present invention;

[0035] Figure 2 It is a schematic diagram of the calculation sequence of CuTree of miniGOP with I3BP structure under single viewpoint in the prior art;

[0036] Figure 3 The calculation order of Cutree of miniGOP of I3BP structure under dual viewpoints in the Cutree technology optimization method of multi-viewpoint video coding in an embodiment of the present invention;

[0037] Figure 4 It is a schematic diagram showing that the inter-frame prediction reference block of the current block in the Cutree technology optimization method for multi-view video coding according to an embodiment of the present invention may cover multiple 16*16 pixel blocks. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] See also Figure 1 , which is a flowchart of a method for optimizing the Cutree technology for multi-view video coding according to an embodiment of the present invention, comprising the following steps:

[0040] S1, based on the pre-analysis lookahead process, select all the frames to be coded from the viewpoints of a GOP into the lookahead sequence, which will also be the sequence analyzed by Cutree. The resolution of these coded frames is reduced to 1 / 2 of the original and stored. The coding blocks are divided into 16*16 pixels and coarse intra-frame prediction, coarse inter-frame prediction and coarse inter-viewpoint prediction are performed. The three costs are recorded as C intra , C inter , and C inter-view , and store it in the data structure of the corresponding frame;

[0041] S2, enter the lookahead sequence again, traverse forward from the last frame, first find the last non-B frame (generally speaking, the last frame of the sequence is a non-B frame), and then find all the viewpoint frames of the non-B frame. The sequence numbers of these frames in the sequence should be continuous; then search forward to find the second non-B frame and all its viewpoint frames. These two non-B frames form a miniGOP. Assume that the number of viewpoints is N, then there are N miniGOPs at this time;

[0042] S3, taking the most commonly used prediction structure P1b1Bb2P2 in engineering as an example, according to the principle of Cutree, frames that are not referenced should be back-propagated first, and frames that are referenced many times should be back-propagated last to obtain the importance of all reference frame propagations. That is, for miniGOP, the calculation order should be b1b2BP2P1, such as Figure 2 As shown. However, it should be noted that when multiple viewpoints are introduced, there is also a reference relationship between viewpoints, which can be abstracted as a reference relationship between different miniGOPs. Such a reference relationship will also affect the calculation order of CuTree. Viewpoints with more references should be calculated later, and viewpoints that are not referenced should be calculated first. To better understand this, take a dual-view video as an example, Figure 3 As shown in the figure, the dual-view video is divided into the main view and the auxiliary view, where the main view is the reference view and the auxiliary view is the reference view. Then for the miniGOP of these two views, in order to meet the calculation principle of Cutree, the calculation order of the frames for the miniGOP of these two views is set to b 辅1 b 主1 b 辅2 b 主2 B 辅 B 主 P 辅2 P 主2 P 辅1 P 主1 ;

[0043] S4, according to the set calculation order, extract the corresponding C of each block in the current frame intra, C inter and C inter-view In Cutree, when calculating the importance, the block traversal is performed in units of 16*16 pixels, and then one of the 16*16 pixel blocks in the current frame is used to calculate the sum of the propagation information propagated to the current block from the other blocks that refer to it. cost , which depends on the amount of information propagated by the block referring to the current block, namely:

[0044]

[0045] Among them, propagate amount _ref i represents the amount of information propagated from the i-th reference current block to the current block; for the i-th reference block, the total amount of information to be propagated is:

[0046] propagate amount =(C intra *invQScales+propagate cost )*Fpropagate

[0047] Among them, invQScales is a fixed value related to the frame-level QP of the frame where the current block is located, and Fpropagate is the ratio of information obtained from the reference frame;

[0048]

[0049] Assume that the current block's C inter C intra 70%, indicating that 30% of the information of the current block comes from its corresponding reference block, F propagate is 0.3.

[0050] S5,When propagating, we also need to consider the matching blocks of inter-frame prediction and inter-view prediction, which may cover multiple blocks in the reference frame, such as Figure 4 At this time, it is necessary to propagate according to the ratio of pixel area. amount In addition, some B frames may use bidirectional prediction when performing inter-frame prediction. In this case, propagate amount The reference blocks are equally divided into two reference frames. After the current block is propagated, the next block is entered for further propagation;

[0051] Considering that propagate is calculated first amount The auxiliary viewpoint b-frame will not be referenced by any other frame, so for the auxiliary viewpoint b-frame, propagate cost is 0, so it will only propagate for its reference frameamount Since it has no importance information, its block-level QP will not be affected by the Cutree technology.

[0052] S6, if the propagate of all blocks of a frame cost After the calculation is completed, the QP of each block is adjusted according to the value to assign different bit rates to different blocks according to their different importance; QP adjustment value QP offset The calculation formula is:

[0053]

[0054] Among them, strength is the adjustment strength set in the Cutree technology, which is generally set to 2; each block is based on the calculated QP offset Adjust QP to reasonably allocate bit rate and improve the subjective and objective quality of the video.

[0055] Specifically, in S1, coarse intra-frame prediction means that compared with conventional intra-frame prediction, only a small number of prediction modes' SATD are calculated, including 8 types, namely DC, PLANAR, ANGLE5, 10, 15, 20, 25 and 30. Then, the mode with the best SATD is selected to perform SATD calculations for two prediction angles with a step size of 2. Finally, the best SATD is selected as the cost and its corresponding prediction mode is used as the optimal prediction angle of the coarse intra-frame prediction mode. Coarse inter-frame prediction refers to performing motion search in the reduced reference frame to find the best matching block, and calculating the SATD value of the residual as the cost of coarse inter-frame prediction. Coarse inter-viewpoint prediction refers to performing motion search in the referenced viewpoint frame at the same POC as the reference frame to find the best matching block, and calculating the SATD value of the residual as the cost of coarse inter-viewpoint prediction.

[0056] Furthermore, in S4, for the frame of the main viewpoint, since inter-view prediction is not performed, Fpropagate is only calculated by C intra , C inter Perform calculations;

[0057]

[0058] Assume that the current block's C inter C intra 70%, indicating that 30% of the information of the current block comes from its corresponding reference block, F propagate is 0.3.

[0059] For frames from non-main viewpoints, we need to further consider the impact of inter-viewpoint prediction. To calculate Fpropagate, we first calculate C inter With C inter-viewThe difference ratio ΔC (%):

[0060] If C inter and C inter-view If the difference ratio ΔC (%) exceeds the given threshold Sh, the minimum value of the two is taken to calculate the ratio; if it does not exceed the given threshold, it is considered that the current frame must transmit a certain proportion of information for its main viewpoint and the temporal reference frame, and the proportion value depends on the frame type of the main viewpoint and the POC distance of the temporal reference; if the frame type of the main viewpoint frame referenced by the current frame is a key frame, the transmission ratio value Fpropagate is 90%; if it is a P frame, the transmission ratio Fpropagate is 70%, and if it is a B frame, the transmission ratio Fpropagate is 50%; if the POC distance of the temporal reference frame of the current frame is greater than 4, the transmission ratio Fpropagate is 50%, otherwise the transmission ratio is 70%. Based on the size relationship between the difference ratio and the given threshold Sh, as well as the reference main viewpoint frame type and the POC distance of the temporal reference, the formula for taking the value of Fpropagate is expressed as follows:

[0061]

[0062] The threshold Sh is fixed at 0.4.

[0063] In a specific application example, in order to reflect the effectiveness of the method of the embodiment of the present invention, Table 1 shows the performance change of using the Cutree technology before optimization and the Cutree technology after optimization compared with the original encoding under three 1080p and three 720p 3D sequence tests, expressed as ΔBD-Rate (the larger the negative value, the better the quality). It can be seen from the table that the performance of the Cutree technology after optimization on each test sequence is better than that of the Cutree technology before optimization, and the encoding bitstream size can be reduced by about 4% at the same quality.

[0064] Table 1 Performance changes of original coding before and after optimization using Cutree technology

[0065]

[0066] It should be understood that the exemplary embodiments described herein are illustrative rather than restrictive. Although one or more embodiments of the present invention have been described in conjunction with the accompanying drawings, it should be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A method for optimizing the Cutree technology for multi-view video coding, characterized in that: The following steps are involved: S1, based on the pre-analysis lookahead process, select all the frames to be coded from the viewpoints of a GOP into the lookahead sequence, reduce the resolution of these coded frames to 1 / 2 of the original and store them, divide the coding blocks into 16*16 pixels, perform coarse intra-frame prediction, coarse inter-frame prediction and coarse inter-viewpoint prediction, and the three costs are recorded as C intra , C inter , and C inter-view , and store it in the data structure of the corresponding frame; S2, enter the lookahead sequence again, traverse forward from the last frame, first find the last non-B frame, and then find all the viewpoint frames of the non-B frame. The sequence numbers of these frames in the sequence should be continuous; then search forward to find the second non-B frame and all its viewpoint frames. These two non-B frames form a miniGOP. Assume that the number of viewpoints is N, then there are N miniGOPs at this time; S3, for the miniGOP of these two viewpoints, set the frame calculation order to b 辅1 b 主1 b 辅2 b 主2 B 辅 B 主 P 辅2 P 主2 P 辅1 P 主1 ; S4, according to the set calculation order, extract the corresponding C of each block in the current frame intra , C inter and C inter-view In Cutree, when calculating the importance, the block traversal is performed in units of 16*16 pixels, and then one of the 16*16 pixel blocks in the current frame is used to calculate the sum of the propagation information propagated to the current block from the other blocks that refer to it. cost , which depends on the amount of information propagated by the block referring to the current block, namely: Among them, propagate amount _ref i represents the amount of information propagated from the i-th reference current block to the current block; for the i-th reference block, the total amount of information to be propagated is: propagate amount =(C intra *invQScales+propagate cost )*Fpropagate Among them, invQScales is a fixed value related to the frame-level QP of the frame where the current block is located, and Fpropagate is the ratio of information obtained from the reference frame; S5, propagate according to the ratio of pixel area amount Split and pass them to different covered blocks, and further propagate amount The reference blocks are equally divided into two reference frames. After the current block is propagated, the next block is entered for further propagation. S6, if the propagate of all blocks of a frame cost After the calculation is completed, the QP of each block is adjusted according to the value to assign different bit rates to different blocks according to their different importance; QP adjustment value QP offset The calculation formula is: Among them, strength is the adjustment strength set in the Cutree technology, which is generally set to 2; each block is based on the calculated QP offset Adjust QP to reasonably allocate bit rate and improve the subjective and objective quality of the video.

2. The Cutree technology optimization method for multi-view video coding according to claim 1, characterized in that: In S1, the coarse intra-frame prediction means that compared with the conventional intra-frame prediction, only a small number of prediction modes of SATD are calculated, including 8 types including DC, PLANAR, ANGLE5, 10, 15, 20, 25 and 30. Then, the mode with the best SATD is selected to perform SATD calculations for two prediction angles with a step size of 2. Finally, the best SATD is selected as the cost and its corresponding prediction mode is selected as the optimal prediction angle of the coarse intra-frame prediction mode.

3. The Cutree technology optimization method for multi-view video coding according to claim 1, characterized in that: In S1, the coarse inter-frame prediction refers to performing motion search in the reduced reference frame to find the best matching block, and calculating the SATD value of the residual as the cost of the coarse inter-frame prediction.

4. The Cutree technology optimization method for multi-view video coding according to claim 1, characterized in that: In S1, the coarse inter-viewpoint prediction refers to performing motion search on the referenced viewpoint frame in the same POC as a reference frame to find the best matching block, and calculating the SATD value of the residual as the cost of the coarse inter-viewpoint prediction.

5. The Cutree technology optimization method for multi-view video coding according to any one of claims 1 to 4, characterized in that: For the main viewpoint frame, since no inter-view prediction is performed, Fpropagate is only calculated by C intra , C inter Perform calculations; 6. The Cutree technology optimization method for multi-view video coding according to any one of claims 1 to 4, characterized in that: For frames from non-primary viewpoints, we need to further consider the impact of inter-viewpoint prediction. To calculate Fpropagate, we first calculate C inter With C inter-view The difference ratio △C (%): Based on the relationship between the difference ratio and the given threshold Sh, as well as the referenced main viewpoint frame type and the POC distance of the temporal reference, the value of Fpropagate is as follows: The threshold Sh is fixed at 0.4.

Images