An efficient depth map encoding method based on complexity hierarchical annotation

Abstract

The present invention discloses a method for efficient coding of depth maps based on complexity hierarchical annotation, and belongs to the technical field of depth map coding in three-dimensional videos. The technical problem of high complexity of intra-frame coding of depth maps in three-dimensional videos is solved. The technical scheme is as follows: it includes the following steps: S1, complexity hierarchical annotation of coding units of depth maps based on texture and depth features of three-dimensional videos, and establishment of data sets; S2, construction of a depth map complexity hierarchical division network, training the network using the established data sets, and realizing complexity grading of depth map coding units without pre-coding information; S3, formulation of a flow chart of an efficient intra-frame coding method for depth maps based on a depth map complexity hierarchical division network. The beneficial effects of the present invention are: the intra-frame coding complexity of depth maps is reduced under the same depth map reconstruction quality, the present invention uses existing coding information to complete hierarchical annotation, and batch annotation is possible, and has the potential to be promoted in practical engineering applications.

Description

Technical Field

[0001] The present invention relates to the technical field of depth map coding in three-dimensional video, and in particular to an efficient depth map coding method based on complexity graded annotation. Background Art

[0002] The depth map represents the distance between the object and the camera. It is mainly composed of large smooth areas with similar pixel values ​​and edge areas with sudden changes in pixel values. Coding distortion in the edge area will cause holes or boundary oscillations in the synthesized virtual viewpoint. In order to ensure that the synthesized virtual viewpoint does not have obvious distortion, it is necessary to ensure the coding quality of the edge area of ​​the depth map. To this end, 3D-HEVC introduces the depth modeling mode (DMM) based on the existing 35 HEVC intra-frame prediction modes, mainly DMM1 and DMM4. However, due to the uncertainty and diversity of DMM when dividing the current coding area, this will bring huge computational complexity to the intra-frame prediction coding of the depth map.

[0003] At present, the research on efficient depth map coding technology mainly focuses on two aspects:

[0004] First, the study of efficient coding technology for depth maps that are different from texture maps. Since the depth map is used to represent the distance between the object and the camera in the scene, the edge information in the depth map is more important than the edge information in the texture map. When reducing the complexity of depth map encoding, it is necessary to accurately retain the edge information of the depth map, thereby ensuring the accuracy of depth map encoding. In order to reduce the encoding complexity of the depth map while maintaining the quality of depth map encoding, it is necessary to encode the depth map according to the edge characteristics of the depth map, such as using predictions based on plane segmentation, wedge segmentation, and contour segmentation. However, this type of method mainly relies on the setting of thresholds. Fine-grained edges can be detected through artificial threshold settings. The coarser judgment granularity makes the algorithm insensitive, the threshold setting is difficult, and the accuracy is limited. It is difficult to reduce the complexity of depth map encoding while ensuring the accuracy of depth map encoding.

[0005] Second, research on fast encoding technology of depth maps based on synthetic viewpoint quality. In ordinary 3D video formats, depth maps are used to synthesize virtual viewpoints, and depth map encoding distortion will directly lead to virtual viewpoint distortion. Therefore, the fast encoding strategy of depth maps needs to consider synthetic viewpoint distortion. At present, many scholars at home and abroad have also carried out a lot of research on the calculation method of synthetic viewpoint distortion, such as synthetic viewpoint distortion calculation method based on depth image based rendering (DIBR) technology and virtual viewpoint distortion prediction method based on synthetic viewpoint distortion model. However, such methods only guide depth map encoding according to the quality of virtual viewpoint synthesis. For depth maps that do not use synthetic viewpoints, such methods cannot improve their coding efficiency.

[0006] With the rise of artificial intelligence in recent years, deep learning technology has achieved remarkable results in a large number of visual tasks with its powerful feature extraction capabilities. The convolutional neural network model has made breakthrough progress in video feature extraction compared to traditional methods. However, the lack of data samples for depth map annotation and the limited ability to express single-scale features are the bottlenecks that limit the application of deep learning technology to efficient depth map encoding. Summary of the invention

[0007] The purpose of the present invention is to provide an efficient depth map encoding method based on complexity hierarchical labeling to solve the technical problem of high complexity of intra-frame encoding of depth maps in three-dimensional videos. The intra-frame encoding complexity of depth maps is reduced under the same depth map reconstruction quality. In addition, the present invention uses existing coding information to complete hierarchical labeling and can be batch labeled, which has the potential to be promoted in practical engineering applications.

[0008] The inventive idea of ​​the present invention is: the present invention provides an efficient depth map encoding method based on complexity hierarchical annotation. First, the coding units of the depth map are labeled with complexity hierarchical labels based on the texture and depth features of the three-dimensional video. Then, a data set is established. After that, a depth map complexity hierarchical partitioning network is constructed. The network is trained using the established data set. Without the need for pre-coding information, complexity classification of the depth map coding units is achieved. Finally, a flow chart of the efficient intra-frame encoding method for the depth map based on the depth map complexity hierarchical partitioning network is formulated.

[0009] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically: a method for efficiently encoding a depth map based on complexity hierarchical annotation, comprising the following steps:

[0010] 1.1. Based on the texture and depth features of 3D video, the coding units with a size of 64×64 in the depth map are labeled in a hierarchical manner to establish a dataset;

[0011] 1.2. Construct a depth map complexity classification network, use the established data set to train the network, and realize the complexity classification of the depth map coding unit without the need for precoding information;

[0012] 1.3. Develop a flow chart of the efficient intra-frame coding method for depth maps based on a depth map complexity classification network.

[0013] Furthermore, the step 1.1 specifically includes the following steps:

[0014] 2.1. Obtain 3D video from the official test platform, perform intra-frame coding on the texture map of the 3D video, and record the prediction mode of the prediction unit;

[0015] 2.2. The prediction modes of the texture map intra prediction unit are divided into three categories and described by the texture density parameter T as the texture features of the 3D video texture map:

[0016]

[0017] 2.3. Intra-frame encoding of the depth map of the 3D video and recording the prediction mode of the prediction unit;

[0018] 2.4. The prediction modes of the depth map intra prediction unit are divided into three categories and described by the depth density parameter D as the depth feature of the 3D video depth map:

[0019]

[0020] 2.5. Based on texture and depth features, the complexity level L of the coding unit with a size of 64×64 in the depth map is described as:

[0021]

[0022] Wherein, M represents the number of prediction units included in the depth map coding unit, and N represents the number of prediction units included in the texture map coding unit at the position corresponding to the depth map coding unit;

[0023] 2.6. The coding unit of size 64×64 in the texture map and the coding unit of size 64×64 in the depth map are used as the input image pair for network training, the complexity level L of the coding unit of size 64×64 in the depth map corresponding to the input image pair is used as the true label, and the input image pair and its corresponding true label are taken as a sample;

[0024] 2.7. Use the holdout method to take 80% of all data samples as the training set and the remaining 20% ​​as the test set.

[0025] Furthermore, the step 1.2 specifically includes the following steps:

[0026] 3.1. The coding unit with a size of 64×64 in the texture image and the coding unit with a size of 64×64 in the depth image are used as the input image of the network, the number of output channels is 3, and the same filling mode is used in the training process;

[0027] 3.2. Perform two sets of convolutions with a kernel of 3×3, a channel of 32, and a step size of 1 on the images input to the depth map feature extraction module and the texture map feature extraction module to obtain local features of the depth image and the texture image;

[0028] 3.3. Perform 8×8, 4×4 and 2×2 maximum pooling on the local features of the obtained depth image and texture image respectively to obtain hierarchical depth and texture features at three levels;

[0029] 3.4. For the layered depth and texture features, two groups of convolutions with a kernel of 3×3, 64 channels, and a stride of 1 are first performed, and then the features are concatenated to form a combined feature;

[0030] 3.5. Perform two-layer full connection on the combined features, and then output the results of complexity classification annotation.

[0031] Furthermore, the step 1.3 specifically includes the following steps:

[0032] 4.1. Preprocess the 3D video to be encoded, call the depth map complexity classification network, and record the complexity level of the depth map encoding unit;

[0033] 4.2. Start intra-frame prediction, read the current coded frame, and determine whether the current video frame to be encoded is a depth map;

[0034] 4.3. For the depth map, read the complexity level of the corresponding coding unit position;

[0035] 4.4. For coding units with complexity level 0, the rate-distortion cost is recursively calculated between the depth range [0-1]. When calculating the rate-distortion cost, the mode range of the prediction unit is traversed and selected between Intra_Planar0 and Intra_DC1;

[0036] 4.5. For coding units with complexity level 1, the rate-distortion cost is recursively calculated between the depth range [1-2]. When calculating the rate-distortion cost, the mode range of the prediction unit is traversed and selected between Intra_Angular2 and 34;

[0037] 4.6. For coding units with complexity level 2, the rate-distortion cost is recursively calculated between the depth range [2-3]. When calculating the rate-distortion cost, the mode range of the prediction unit is traversed and selected between DMM1 and DMM4.

[0038] 4.7. For the texture map, perform the conventional intra-frame prediction process;

[0039] 4.8. Quantize, filter and entropy encode the texture map and depth map after intra-frame prediction, and output a three-dimensional video code stream.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) The present invention takes reducing the complexity of intra-frame prediction coding of depth maps as a starting point, introduces deep learning technology into the 3D video coding process, and reasonably simplifies the candidate mode set of intra-frame prediction of depth maps;

[0042] (2) The present invention utilizes the existing 3D video coding information and relies on the texture map and depth map features to grade and annotate the complexity of the depth map coding unit. The method can be batch annotated and has the potential to be promoted in practical engineering applications.

[0043] (3) The present invention constructs a depth map complexity classification network, by extracting texture maps and depth Figure 3 The hierarchical features at different scales break through the limitation of single-scale feature expression ability and realize the complexity classification of depth map coding unit without pre-coding information.

[0044] (4) The present invention reduces the complexity of intra-frame coding of the depth map under the same depth map reconstruction quality. Compared with the 3D-HEVC standard test model HTM16.0, it can accelerate the depth map coding process while ensuring the quality of the depth map. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0046] Figure 1 A flowchart of a method for efficiently encoding a depth map based on complexity classification annotation provided by the present invention.

[0047] Figure 2 Schematic diagram of the depth graph complexity classification network constructed for the present invention.

[0048] Figure 3 Flowchart of the method for efficient intra-frame encoding of depth maps based on a depth map complexity hierarchical partitioning network formulated for the present invention.

[0049] Figure 4It is a schematic diagram of the comparison results of the subjective quality of the Undo_Dancer video sequence between the present invention and the 3D-HEVC standard test model HTM16.0; wherein, (a) is a schematic diagram of the video reconstruction results of the Undo_Dancer video sequence under the original HTM16.0 model; (b) is a schematic diagram of the video reconstruction results of the Undo_Dancer video sequence using the method of Example 1 of the present invention.

[0050] Figure 5 It is a schematic diagram of the comparison results of the subjective quality of the Poznan_Hall2 video sequence between the present invention and the 3D-HEVC standard test model HTM16.0; wherein, (a) is a schematic diagram of the video reconstruction results of the Poznan_Hall2 video sequence under the original HTM16.0 model; (b) is a schematic diagram of the video reconstruction results of the Poznan_Hall2 video sequence using the method of Example 2 of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0052] Example 1

[0053] See also Figures 1 to 4 The technical solution provided by this embodiment is a method for efficiently encoding a depth map based on complexity classification annotation, comprising the following steps:

[0054] 1. Based on the texture and depth features of 3D video, the coding units with a size of 64×64 in the depth map are labeled in a hierarchical manner to establish a dataset;

[0055] 2. Construct a depth map complexity classification network, use the established data set to train the network, and realize the complexity classification of the depth map coding unit without the need for pre-coding information;

[0056] 3. Develop a flow chart for the efficient intra-frame encoding method of depth maps based on the depth map complexity classification network.

[0057] Specifically, in step 1, based on the texture and depth features of the three-dimensional video, the coding units with a size of 64×64 in the depth map are labeled in a hierarchical manner to establish a data set, including the following steps:

[0058] 1) Obtain 3D video from the official test platform, perform intra-frame coding on the texture map of the 3D video, and record the prediction mode of the prediction unit;

[0059] 2) The prediction modes of the texture map intra prediction unit are divided into three categories and described by the texture density parameter T as the texture features of the 3D video texture map:

[0060]

[0061] 3) Perform intra-frame coding on the depth map of the 3D video and record the prediction mode of the prediction unit;

[0062] 4) The prediction modes of the depth map intra prediction unit are divided into three categories and described by the depth density parameter D as the depth feature of the 3D video depth map:

[0063]

[0064] 5) Based on texture and depth features, the complexity level L of a coding unit of size 64×64 in the depth map is described as:

[0065]

[0066] Wherein, M represents the number of prediction units included in the depth map coding unit, and N represents the number of prediction units included in the texture map coding unit at the position corresponding to the depth map coding unit;

[0067] 6) The coding unit of size 64×64 in the texture map and the coding unit of size 64×64 in the depth map are used as the input image pair for network training, the complexity level L of the coding unit of size 64×64 in the depth map corresponding to the input image pair is used as the true label, and the input image pair and its corresponding true label are used as a sample;

[0068] 7) Use the holdout method to take 80% of all data samples as the training set and the remaining 20% ​​as the test set.

[0069] Specifically, in step 2, refer to Figure 2 , construct a depth map complexity classification network, use the established data set to train the network, and realize the complexity classification of the depth map coding unit without precoding information, including the following steps:

[0070] 1) The coding unit with a size of 64×64 in the texture image and the coding unit with a size of 64×64 in the depth image are used as the input image of the network, the number of output channels is 3, and the same mode is used during training;

[0071] 2) Perform two sets of convolutions with a kernel of 3×3, a channel of 32, and a step size of 1 on the images input into the depth map feature extraction module and the texture map feature extraction module to obtain local features of the depth image and the texture image;

[0072] 3) Perform 8×8, 4×4 and 2×2 maximum pooling on the local features of the obtained depth image and texture image respectively to obtain hierarchical depth and texture features at three levels;

[0073] 4) The layered depth and texture features are first subjected to two sets of convolutions with a kernel of 3×3, 64 channels, and a stride of 1, and then feature concatenation is performed to form a combined feature;

[0074] 5) Perform two-layer full connection on the combined features, and then output the results of complexity classification annotation.

[0075] Specifically, in step 3, refer to Figure 3 , a flow chart of an efficient intra-frame coding method for a depth map based on a depth map complexity hierarchical partitioning network is formulated, comprising the following steps:

[0076] 1) Preprocess the 3D video to be encoded, call the depth map complexity classification network, and record the complexity level of the depth map encoding unit;

[0077] 2) Start intra-frame prediction, read the current coded frame, and determine whether the current video frame to be encoded is a depth map;

[0078] 3) For the depth map, read the complexity level of the corresponding coding unit position;

[0079] 4) For coding units with complexity level 0, the rate-distortion cost is recursively calculated between the depth range [0-1]. When calculating the rate-distortion cost, the mode range of the prediction unit is traversed and selected between Intra_Planar0 and Intra_DC1;

[0080] 5) For coding units with complexity level 1, the rate-distortion cost is recursively calculated between the depth range [1-2]. When calculating the rate-distortion cost, the mode range of the prediction unit is traversed and selected between Intra_Angular2 and 34;

[0081] 6) For coding units with complexity level 2, the rate-distortion cost is recursively calculated between the depth range [2-3]. When calculating the rate-distortion cost, the mode range of the prediction unit is traversed and selected between DMM1 and DMM4;

[0082] 7) For the texture map, perform the conventional intra-frame prediction process;

[0083] 8) Quantize, filter and entropy encode the texture map and depth map after intra-frame prediction, and output a three-dimensional video code stream.

[0084] In order to test the performance of the method proposed in this embodiment, the method of this embodiment is compared with the 3D-HEVC standard test model HTM16.0 from the perspectives of objective performance indicators (time saving and BD-Rate) and subjective visual quality. The test sequence is Undo_Dancer in the standard test sequence. Table 1 shows the experimental comparison results of objective performance indicators.

[0085] Table 1

[0086]

[0087] As shown in Table 1, from the perspective of objective performance indicators, the method of this embodiment can save an average of 37.92% of computational complexity compared to HTM-16.0, and the BD-Rate only increases by 0.26%. Figure 4 As shown, from the perspective of subjective visual quality, at viewpoint 5, frame 20 of the Undo_Dancer video sequence, by comparing the subjective quality of the reconstructed depth map and its detail areas, it can be seen that the depth map reconstructed using the method of this embodiment is smoother in the edge detail areas.

[0088] Example 2

[0089] Based on Example 1, the test sequence is Poznan_Hall2 in the standard test sequence, and Table 2 shows the experimental comparison results of objective performance indicators. As shown in Table 2, from the perspective of objective performance indicators, the method of this embodiment can save an average of 35.70% of the computational complexity compared to HTM-16.0, and the BD-Rate only increases by 0.27%. Figure 5 As shown, from the perspective of subjective visual quality, viewpoint 5, frame 25 of the Poznan_Hall2 video sequence, by comparing the subjective quality of the reconstructed depth map and its detail areas, it can be seen that the method of this embodiment has a better protection effect on the texture edge of the reconstructed depth map.

[0090] The above embodiments all prove that, compared with the 3D-HEVC standard test model HTM16.0, the present invention can accelerate the depth map encoding process while ensuring the quality of the depth map, that is, reduce the complexity of depth map intra-frame encoding under the same depth map reconstruction quality.

[0091] Table 2

[0092]

[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for efficient depth map encoding based on complexity hierarchical annotation, characterized in that: The following steps are involved: S1. Based on the texture and depth features of the three-dimensional video, the coding units in the depth map are labeled in a hierarchical manner to establish a data set; The step S1 comprises the following steps: S11, obtaining a 3D video from an official test platform, performing intra-frame coding on a texture map of the 3D video, and recording a prediction mode of a prediction unit; S12, the prediction modes of the texture map intra prediction unit are divided into three categories, which are described by the texture density parameter T as the texture features of the 3D video texture map: S13, performing intra-frame coding on the depth map of the three-dimensional video, and recording a prediction mode of the prediction unit; S14, the prediction modes of the depth map intra prediction unit are divided into three categories, which are described by the depth density parameter D as the depth feature of the 3D video depth map: S15. Based on texture and depth features, the complexity level L of the coding unit in the depth map is described as: Wherein, M represents the number of prediction units included in the depth map coding unit, and N represents the number of prediction units included in the texture map coding unit at the position corresponding to the depth map coding unit; S16, using the coding unit in the texture map and the coding unit in the depth map as an input image pair for network training, using the complexity level L of the coding unit in the depth map corresponding to the input image pair as a true label, and using the input image pair and its corresponding true label as a sample; S17, use the holdout method to divide all data samples into training set and test set; S2. Construct a depth map complexity classification network, use the established data set to train the network, and achieve complexity classification of depth map coding units without precoding information. The step S2 comprises the following steps: S21, using the coding unit in the texture map and the coding unit in the depth map as the input image of the network, the number of output channels is the number of complexity levels, and the same mode is used during training; S22, performing two sets of convolutions on the images input into the depth map feature extraction module and the texture map feature extraction module, respectively, to obtain local features of the depth image and the texture image; S23, performing maximum pooling at different scales on the local features of the obtained depth image and texture image, respectively, to obtain hierarchical depth and texture features at three levels; S24, performing two groups of convolutions on the layered depth and texture features respectively, and then concatenating the features to form a combined feature; S25, perform two-layer full connection on the combined features, and then output the result of complexity classification labeling; Step S3 includes the following steps: S31, preprocessing the 3D video to be encoded, calling the depth map complexity classification network, and recording the complexity level of the depth map encoding unit; S32, start intra-frame prediction, read the current coded frame, and determine whether the current video frame to be encoded is a depth map; S33, for the depth map, read the complexity level of the corresponding coding unit position; S34, for a coding unit with a complexity level of 0, recursively calculating a rate-distortion cost between a depth range of [0-1], and when calculating the rate-distortion cost, traversing and selecting a mode range of the prediction unit between Intra_Planar0 and Intra_DC1; S35, for a coding unit with a complexity level of 1, recursively calculating a rate-distortion cost between a depth range of [1-2], and when calculating the rate-distortion cost, traversing and selecting a mode range of the prediction unit between Intra_Angular2 and 34; S36, for a coding unit with a complexity level of 2, recursively calculating a rate-distortion cost between a depth range of [2-3], and when calculating the rate-distortion cost, traversing and selecting a mode range of the prediction unit between DMM1 and DMM4; S37, performing a conventional intra-frame prediction process on the texture map; S38, quantizing, filtering and entropy encoding the texture map and depth map after intra-frame prediction, and outputting a three-dimensional video code stream.

Images