Optimization method of cutree technology of multi-view video coding

Abstract

The application discloses a Cutree technology optimization method for multi-view video coding, comprising the following steps: S1, based on a pre-analysis lookahead process, selecting all to-be-coded frames of all viewpoints in a GOP quantity to enter a lookahead sequence, reducing the resolution of the to-be-coded frames to 1 / 2 of the original storage, dividing the coded blocks in units of 16*16 pixels, and performing coarse intra-frame prediction, coarse inter-frame prediction and coarse viewpoint prediction; S2, two non-B frames form a miniGOP, and if the viewpoint quantity is N, then there are N miniGOPs at this time; S3, for the miniGOPs of the two viewpoints, the calculation order of the frames is set as b 辅1 b 主1 b 辅2 b 主2 B 辅 B 主 P 辅2 P 主2 P 辅1 P 主1 ; S4, calculating the total propagate information sum propagate cost of a current block from the rest of the blocks that reference the current block to the current block; S5, after the current block is propagated, the next block is continued to be propagated; and S6, if the propagate cost of all blocks of a frame is calculated, then the QP of each block is adjusted according to the value, so that different code rates are allocated to different blocks according to different importance.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video coding, and particularly relates to a Cutree technology optimization method for multi-view video coding. BACKGROUND

[0002] In recent years, with the continuous improvement of video processing technology and imaging display technology, the demand for video content is steadily developing towards high definition and stereoscopic. For high definition, the video coding expert group of the International Telecommunication Union Telecommunication Standardization Organization and the Moving Picture Experts Group of the International Standardization Organization jointly promote the AVC, HEVC and other standards for efficient compression of high-definition video. For stereoscopic, the standardization organization promotes the corresponding extension standards: MVC and MV-HEVC on the basis of the above standards, which are used for compression of multi-view video content mainly based on binocular video. These video contents can be used as the representation form of most 3D videos to bring the roaming experience of different perspectives to the viewers.

[0003] However, these extension standards only make adaptive modifications for multi-view video compression in the high-level syntax elements and frame-level, but do not make any underlying changes to the originally existing part of the technology for single perspective, which leads to some problems when directly reusing these technologies in the multi-view video standard.

[0004] The Cutree technology is an effective technology for improving the subjective and objective quality of the rate control module under the HEVC standard. The principle is to adjust the QP of the to-be-encoded block according to the reference degree of the to-be-encoded block offset , so as to reasonably allocate the bits of each encoded block. If the to-be-encoded block contributes more information to the subsequent frame, it is considered that the importance of the block is higher, and the encoding quality of the block needs to be improved, so the QP offset is set to a negative value to reduce the QP. On the contrary, the QP offset is set to a positive value to increase the QP. The reference degree described in the principle is obtained based on the encoding lookahead process. If the I frame is referenced by the P frame and multiple B frames, the reference degree of the I frame is very high, and the I frame is given a lower QP. The non-reference B frame does not participate in the adjustment of the Cutree because it has no contribution, that is, the Cutree technology only acts on the frames that can be used as references.

[0005] Multi-view video refers to video recorded by multiple cameras simultaneously from different angles. In the multi-view video coding framework, the reference relationship is more complex than single-path video coding. In addition to inter-frame prediction, inter-view prediction is also added: under the same POC, the frames of the main view can be predicted by the auxiliary view. The increase of the reference relationship makes the original Cutree technology have certain errors when applied. For example, when the inter-frame prediction and inter-view prediction obtained in the lookahead process are very close in cost, it is difficult to accurately determine which mode is better. At this time, if only one mode is arbitrarily selected, the information of the corresponding reference frame is propagated, which may cause the importance calculation of the reference frame of the other mode to be wrong, and then the code rate allocation of the video is affected. Therefore, for this situation, weight allocation optimization is required. SUMMARY

[0006] In view of the above problems, the present application provides a Cutree technology optimization method for multi-view video coding, which uses pre-analysis and inter-view related information to optimize the Cutree technology for multi-view video coding, so as to further improve the efficiency of multi-view video coding standard compression of multi-view video.

[0007] To solve the above technical problems, the present application adopts the following technical scheme:

[0008] A Cutree technology optimization method for multi-view video coding, comprising the following steps:

[0009] S1, based on the pre-analysis lookahead process, select all the to-be-coded frames of all views in a GOP quantity to enter the lookahead sequence, reduce the resolution of these coded frames to 1 / 2 of the original, divide the coded blocks in units of 16*16 pixels, and perform coarse intra-frame prediction, coarse inter-frame prediction and coarse inter-view prediction, the obtained three costs are respectively denoted as C intra , C inter , and C inter-view , and are stored in the data structure of the corresponding frame;

[0010] S2, enter the lookahead sequence again, traverse from the last frame to the front, first find the non-B frame of the last frame, and then find all the view frames of the non-B frame, the serial numbers of these frames in the sequence should be continuous; then search forward again, find the second non-B frame and all its view frames, the two non-B frames form a miniGOP, and the number of views is N, so there are N miniGOPs at this time;

[0011] S3, for the two view miniGOPs, set the calculation order of the frames as b 辅1 b 主1 b 辅2 b主2 B 辅 B 主 P 辅2 P 主2 P 辅 1P 主1 ;

[0012] S4, according to the set calculation order, extract the corresponding C of each block in the current frame intra , C inter and C inter-view In Cutree, when calculating the importance, the block traversal is performed in units of 16*16 pixels, and then one of the 16*16 pixel blocks in the current frame is used to calculate the total propagation information propagated from the rest of the blocks that refer to it to the current block. cost , which depends on the amount of block propagation information referring to the current block, namely:

[0013]

[0014] Among them, propagate amount _ref i represents the amount of information propagated from the i-th reference current block to the current block; for the i-th reference block, the total amount of information to be propagated is:

[0015] Propagate amount =(C intra *invQScales+propagate cost )*Fpropagate

[0016] Among them, invQScales is a fixed value related to the frame-level QP of the frame where the current block is located, and Fpropagate is the ratio of information obtained from the reference frame;

[0017] S5, propagate according to the ratio of pixel area amount Split and pass them to different covered blocks, and further propagate amount The reference blocks are equally divided into two reference frames. After the current block is propagated, the next block is entered to continue propagation.

[0018] S6, if the propagate of all blocks of a frame cost After the calculation is completed, the QP of each block is adjusted according to the value to assign different bit rates to different blocks according to their importance; QP adjustment value QP offset The calculation formula is:

[0019]

[0020] Wherein, strength is the adjustment strength set in Cutree technology, generally set as 2; each block is adjusted according to the calculated QP offset Adjust the QP to reasonably allocate the code rate and improve the subjective and objective quality of the video.

[0021] In a possible implementation, in S1, the coarse intra prediction refers to calculating the SATD of only a small number of prediction modes, including DC, PLANAR, ANGLE5, 10, 15, 20, 25 and 30, in total 8, compared with the conventional intra prediction, then, selecting the mode with the optimal SATD, performing the SATD calculation of two prediction angles with a step of 2, and finally selecting the optimal SATD as the cost and the corresponding prediction mode as the optimal prediction angle of the coarse intra prediction mode.

[0022] In a possible implementation, in S1, the coarse inter prediction refers to performing the motion search in the reduced reference frame to find the optimal matching block, and calculating the SATD value of the residual as the cost of the coarse inter prediction.

[0023] In a possible implementation, in S1, the coarse inter-view prediction refers to performing the motion search in the reference view frame in the same POC as the reference frame to find the optimal matching block, and calculating the SATD value of the residual as the cost of the coarse inter-view prediction.

[0024] In a possible implementation, for the frame of the main view point, since the inter-view prediction is not performed, the Fpropagate is only calculated by C intra and C inter ;

[0025]

[0026] In a possible implementation, for the frame of the non-main view point, the influence of the inter-view prediction needs to be further considered, and the Fpropagate is calculated first by calculating the difference ratio ΔC(%) between C inter and C inter-view .

[0027]

[0028] Based on the size relationship between the difference ratio and a given threshold Sh, and the type of the reference main view point frame and the POC distance of the temporal reference, the Fpropagate takes the following values:

[0029]

[0030] Wherein, the threshold Sh is fixed as 0.4.

[0031] The method has the following beneficial effects:

[0032] (1) Based on the reference characteristics of multi-view video, the propagation order and calculation order in the Cutree implementation process are reconstructed to conform to the actual encoding order of multi-view video coding, so that the code rate allocation result is more accurate.

[0033] (2) The reverse propagation process of the Cutree technology in multi-view video coding is optimized by using the pre-analysis information of the encoder (including the SATD of 16*16 CU-based intra prediction, inter prediction, mode, etc.) and the similarity information between views. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The step flow chart of the Cutree technology optimization method for multi-view video coding of the embodiment of the present application is shown in the figure.

[0035] Figure 2 The Cutree calculation order diagram of the I3BP structure miniGOP under a single view in the prior art is shown in the figure.

[0036] Figure 3 The Cutree calculation order of the I3BP structure miniGOP under double views in the Cutree technology optimization method for multi-view video coding of the embodiment of the present application is shown in the figure.

[0037] Figure 4 The diagram of the inter prediction reference block of the current block possibly covering multiple 16*16 pixel blocks in the Cutree technology optimization method for multi-view video coding of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0039] Referring to Figure 1 , the step flow chart of the Cutree technology optimization method for multi-view video coding of the embodiment of the present application is shown in the figure, including the following steps:

[0040] S1, based on the pre-analysis lookahead process, select all the to-be-encoded frames of all the viewpoints of a GOP quantity into a lookahead sequence, which will also be the sequence for Cutree analysis, reduce the resolution of these encoded frames to 1 / 2 of the original, divide the coding blocks in units of 16*16 pixels, perform coarse intra prediction, coarse inter prediction and coarse inter-viewpoint prediction, and the three obtained costs are denoted as C intra , C inter , and C inter-view , respectively, and stored in the data structure corresponding to the frame;

[0041] S2, enter the lookahead sequence again and traverse from the last frame to the front, first find the last non-B frame (generally, the last frame of the sequence is a non-B frame), and then find all the viewpoint frames of the non-B frame, the sequence numbers of these frames in the sequence should be continuous; then search forward again, find the second non-B frame and all its viewpoint frames, the two non-B frames form a miniGOP, and there are N miniGOPs at this time if the number of viewpoints is N;

[0042] S3, taking the most commonly used prediction structure P1b1Bb2P2 as an example, according to the principle of Cutree, the frames that are not referenced should be preferentially back-propagated, and the frames that are referenced more frequently should be back-propagated last, in order to obtain the importance of the propagation of all reference frames. That is, for a miniGOP, the calculation order should be b1b2BP2P1, as shown in Figure 2 . However, it should be noted that when multi-viewpoints are introduced, there is also a reference relationship between viewpoints, which can be abstracted as a reference relationship between different miniGOPs. Such a reference relationship will also affect the calculation order of Cutree, and the viewpoints that are referenced more frequently should be calculated later, and the viewpoints that are not referenced should be preferentially calculated. In order to better understand this point, taking a two-viewpoint video as an example, as shown in Figure 3 , the two-viewpoint video is divided into a main viewpoint and an auxiliary viewpoint, wherein the main viewpoint is a referenced viewpoint and the auxiliary viewpoint is a reference viewpoint. Then, for the miniGOPs of the two viewpoints, in order to meet the calculation principle of Cutree, the calculation order of the frames of the two viewpoints is set to be b 辅1 b 主1 b 辅2 b 主2 B 辅 B 主 P 辅2 P 主2 P 辅1 P 主1 ;

[0043] S4, according to the set calculation order, extract the corresponding C intra, C inter and C inter-view In Cutree, the importance is calculated by block traversal in 16*16 pixels, then the propagate cost The value depends on the amount of propagate

[0044]

[0045] propagate amount _ref i represents the amount of propagate

[0046] propagate amount = (C intra *invQScales + propagate cost )*Fpropagate

[0047] Where invQScales is a fixed value related to the frame-level QP of the frame where the current block is located, and Fpropagate is the proportion of information obtained from the reference frame;

[0048]

[0049] Assuming that the C inter of the current block is 70% of C intra , which means that 30% of the information of the current block comes from its corresponding reference block, and F propagate is 0.3.

[0050] S5, when propagating, the matching blocks of inter-frame prediction and inter-view prediction may cover multiple blocks in the reference frame, as shown in Figure 4 At this time, it is necessary to divide propagate amount according to the proportion of pixel area and transmit it to different blocks covered. In addition, part of the B frame may use bidirectional prediction when performing inter-frame prediction, at which time propagate amount needs to be equally divided into the reference blocks of the two reference frames. After the current block is propagated, the next block continues to propagate;

[0051] Considering that the auxiliary view b frame is calculated first propagate amount , it will not be referenced by any other frame, so for the b frame of the auxiliary view, propagate cost is 0, so it will only propagate propagateamount Since it has no importance information, it will not be affected by the Cutree technology either.

[0052] S6, if the propagate of all blocks of a frame is 0, the frame is not affected by the Cutree technology. cost After the calculation, the QP of each block is adjusted according to the value, so as to allocate different code rates to different blocks according to different importance; the QP adjustment value QPadjust is calculated as follows: offset

[0053]

[0054] Wherein, the strength is the adjustment strength set in the Cutree technology, generally set as 2; each block adjusts the QP according to the calculated QP offset to allocate code rates reasonably and improve the subjective and objective quality of the video.

[0055] Specifically, in S1, the coarse intra prediction refers to calculating the SATD of only a small number of prediction modes, including DC, PLANAR, ANGLE5, 10, 15, 20, 25 and 30, a total of 8, compared with the conventional intra prediction, and then selecting the mode with the optimal SATD to calculate the SATD of two prediction angles with a step of 2, and finally selecting the optimal SATD as the cost and the corresponding prediction mode as the optimal prediction angle of the coarse intra prediction mode. The coarse inter prediction refers to performing motion search in the reduced reference frame to find the optimal matching block and calculating the SATD value of the residual as the cost of the coarse inter prediction. The coarse inter-view prediction refers to performing motion search in the reference view frame in the same POC as the reference frame to find the optimal matching block and calculating the SATD value of the residual as the cost of the coarse inter-view prediction.

[0056] Further, in S4, for the frames of the main view point, since no inter-view prediction is performed, the Fpropagate is only calculated by C intra and C inter ;

[0057]

[0058] Suppose that the C inter of the current block is 70% of C intra , which indicates that 30% of the information of the current block comes from the corresponding reference block, and F propagate is 0.3.

[0059] And for the frames of the non-main view point, the influence of the inter-view prediction needs to be further considered, and the Fpropagate is calculated first C inter and C inter-view ​the difference ratio ΔC(%) of C

[0060] If C inter and C inter-view If the difference ratio ΔC(%) of C and C exceeds a given threshold value Sh, the minimum value of the two is taken for the calculation of the proportion; if it does not exceed the given threshold value, it is considered that the current frame transmits a certain proportion of information for both its main viewpoint and the temporal reference frame, and the proportion value depends on the frame type of the main viewpoint and the POC distance of the temporal reference; if the frame type of the main viewpoint frame referenced by the current frame is a key frame, the transmission proportion Fpropagate is 90%; if it is a P frame, the transmission proportion Fpropagate is 70%, and if it is a B frame, the transmission proportion Fpropagate is 50%; if the POC distance of the temporal reference frame of the current frame is greater than 4, the transmission proportion Fpropagate is 50%, otherwise the transmission proportion is 70%. Based on the size relationship between the difference ratio and the given threshold value Sh, and the frame type of the referenced main viewpoint and the POC distance of the temporal reference, the Fpropagate value formula is as follows:

[0061]

[0062] The threshold value Sh is fixed at 0.4.

[0063] In a specific application example, in order to reflect the effectiveness of the method of the embodiment of the application, Table 1 shows the performance change of the original encoding compared with the use of the optimized Cutree technology and the use of the optimized Cutree technology under the test of three 1080p and three 720p 3D sequences, which is represented by ΔBD-Rate (the greater the negative value, the better the quality). It can be seen from the table that the performance of the optimized Cutree technology is better than that of the optimized Cutree technology on each test sequence, and the encoding stream size can be reduced by about 4% at the same quality.

[0064] Table 1 Performance change of the original encoding compared with the use of the optimized Cutree technology and the use of the optimized Cutree technology

[0065]

[0066] It should be understood that the example embodiments described herein are illustrative and non-limiting. Although one or more embodiments of the application are described in conjunction with the attached figures, it should be understood that various changes in form and detail can be made without departing from the spirit and scope of the application as defined by the appended claims.

Claims

1. A method for optimizing Cutree technology of multi-view video coding, characterized in that, The method comprises the following steps: S1, based on a pre-analysis lookahead process, select all the to-be-encoded frames of a number of GOPs of all the viewpoints into a lookahead sequence, reduce the resolution of these encoded frames to 1 / 2 of the original, divide the encoded blocks in units of 16*16 pixels, perform coarse intra-frame prediction, coarse inter-frame prediction and coarse inter-viewpoint prediction, and obtain three kinds of cost, respectively denoted as C intra , C inter , and C inter-view , and store them into the data structure of the corresponding frames; S2, entering the lookahead sequence again, traversing from the last frame, first finding the last non-B frame, and then finding all the view frames of the non-B frame, the serial numbers of these frames in the sequence should be continuous; then searching forward again, finding the second non-B frame and all its view frames, the two non-B frames form a miniGOP, assuming the number of views is N, then there are N miniGOPs at this time; S3, for the miniGOP of the two viewpoints, set the calculation order of the frame as b 辅1 b 主1 b 辅2 b 主2 B 辅 B 主 P 辅2 P 主2 P 辅1 P 主1 ; S4, according to the set calculation order, extract the corresponding C intra , C inter and C inter-view In Cutree, when calculating the importance, the block is traversed in 16*16 pixel units, and then the propagate cost value is calculated according to the amount of propagation information of the blocks referring to the current block, that is: wherein propagate amount _ref i represents the amount of propagated information from the i-th reference current block to the current block; and the total amount of information to be propagated for the i-th reference block is: propagate amount = (C intra *invQScales + propagate cost ) * Fpropagate Wherein, invQScales is a fixed value related to the frame-level QP of the frame where the current block is located, and Fpropagate is the proportion of information obtained from the reference frame; S5, propagate according to the proportion of pixel area amount Split, respectively, to cover the different blocks, further propagate amount Divide the reference block of two reference frames, and the current block is propagated. After the completion of the block, continue to propagate the next block; S6, if the propagate of all blocks in a frame is true cost QP adjustment value QP is calculated according to the value, so as to assign different code rates to different blocks according to different importance; the calculation formula of the QP adjustment value QP is as follows: offset QP adjustment value QP is calculated according to the value, so as to assign different code rates to different blocks according to different importance; the calculation formula of the QP adjustment value QP is as follows: Wherein, strength is the adjustment strength set in Cutree technology, which is set to 2; each block is adjusted according to the calculated QP offset Adjust the QP to reasonably allocate the code rate and improve the subjective and objective quality of the video.

2. The method of claim 1, wherein the Cutree technique optimization is performed for each view of the multi-view video coding. In S1, the coarse intra prediction refers to calculating the SATD of only a small number of prediction modes, including DC, PLANAR, ANGLE5, 10, 15, 20, 25 and 30, a total of 8, then selecting the mode with the optimal SATD, and then calculating the SATD of two prediction angles with a step of 2, and finally selecting the optimal SATD as the cost and the corresponding prediction mode as the optimal prediction angle of the coarse intra prediction mode.

3. The method of claim 1, wherein the Cutree technique optimization is performed for each view of the multi-view video coding. In S1, the coarse inter prediction refers to performing motion search in the reduced reference frame to find the optimal matching block, and calculating the SATD value of the residual as the cost of the coarse inter prediction.

4. The method of claim 1, wherein the Cutree technique optimization is performed for each view of the multi-view video coding. In S1, the coarse view prediction refers to performing motion search in the reference view frame in the same POC as the reference frame to find the optimal matching block, and calculating the SATD value of the residual as the cost of the coarse view prediction.

5. The method of claim 1 to 4, wherein, For the frames of the main view, Fpropagate is calculated only through C intra , C inter , since no inter-view prediction is performed.

6. The method of claim 1 to 4, wherein the Cutree technique optimization method of multi-view video coding is characterized by, For the frames of non-main view, the influence of inter-view prediction needs to be considered further to calculate Fpropagate inter The difference between C inter-view and C is ΔC (%): Based on the size relationship of the difference ratio and the given threshold Sh, and the type of the reference main view frame and the POC distance of the temporal reference, Fpropagate takes the following values: Wherein, the threshold Sh is fixed at 0.4.

Images