3D-HEVC depth map fast coding method based on DCAE model

Abstract

The invention provides a 3D-HEVC depth map fast coding method based on a DCAE model, and the method comprises the steps: constructing a data set through employing a 3D video sequence, coding the video sequence, and enabling the coded video sequence to serve as a real label to form a training set; calculating a CTU complexity parameter and a normalized QP value of each coding tree unit; a dual-channel attention enhancement model is constructed, the dual-channel attention enhancement model comprises two channels, each channel is provided with a convolution feature extraction module, and the convolution feature extraction modules of the two channels are connected with an attention mechanism module. The CTU complexity parameter and the normalized QP value are spliced with the output features of the attention mechanism module and then are input into a full connection layer; training a dual-channel attention enhancement model by using the training set; and inputting the test sequence into the trained DCAE model, and judging whether the CU continues to be divided or not according to the output prediction probability and the decision-making mode. According to the method, the coding efficiency is remarkably improved while the video quality is maintained, the coding complexity is effectively reduced, and powerful support is provided for the development of a 3D video coding technology.

Description

Technical Field

[0001] The present invention relates to the technical field of video coding, and in particular to a 3D-HEVC depth map fast coding method. Background Art

[0002] With the continuous development of media technology, 3D-HEVC has emerged as a new generation of video compression standard. While retaining the high compression efficiency of HEVC, 3D-HEVC is optimized for 3D video and can better handle stereoscopic and multi-view video content. However, traditional video coding technology is not fully compatible with depth map data. The new coding tools introduced by 3D-HEVC to improve the efficiency of depth map coding have improved the coding effect, but also led to a significant increase in coding time and computational complexity. This has become a key challenge for the widespread application of 3D-HEVC. Therefore, it is crucial to reduce computational complexity while ensuring coding performance.

[0003] In the research field of 3D-HEVC related technologies, many scholars have carried out work from different angles, aiming to improve coding efficiency while maintaining video quality. They can be summarized into two directions. The first is the improvement starting from CU (coding unit) division: L.Shen et al. proposed a fast TU (transform unit) size decision algorithm for HEVC encoder, which uses Bayesian theorem detection to improve coding efficiency by optimizing TU size decision. Bakkouri et al. proposed different methods to optimize CU division. First, they proposed an early termination method for CU division for 3D-HEVC inter-coding based on boosting neural network; and by building a neural network model, they collected and analyzed a large amount of coding data features, and terminated the CU division process in advance when the data met the conditions set by the model. For 3D-HEVC inter-coding, the team developed a fast CU size and mode decision algorithm, which comprehensively considered a variety of coding-related factors, and based on these factors, targeted optimization and adjustment of the CU size in different situations. In later studies, a fast CU size decision algorithm for 3D-HEVC inter-coding based on machine learning was proposed, which made full use of the advantages of machine learning technology to accurately determine the CU size, making the encoding process more efficient. Yang et al. used unsupervised learning algorithms to mine the implicit intrinsic patterns in depth map data, which reflect the characteristics of different areas of the image; then dynamically adjusted the CU size based on these patterns, making the CU division more suitable for the image content, thereby optimizing the entire encoding process. Wang creatively applied IoT data to the decision-making of 3D-HEVC fast encoding, extracted information of great value for CU division, and optimized the CU division strategy based on this, and finally achieved fast encoding. Zou et al. implemented fast depth map encoding of 3D-HEVC based on Bayesian decision theorem. In the CU division stage, they made full use of Bayesian decision theory to make reasonable division decisions that conform to the actual situation of the image, effectively improving the encoding efficiency. In their research, Zhang et al. also proposed the optimization of CU division. By analyzing the characteristics of the depth map, they determined a method that can quickly adjust the CU size. This method takes into account the local and global characteristics of the depth map, making the CU division more reasonable while ensuring the encoding quality, which helps to improve the encoding speed.

[0004] Second, the mode selection is improved: Lei et al. proposed a fast mode decision for multi-view deep video coding based on the correlation information between different views and components, established a correlation model, and quickly selected the appropriate coding mode based on this model, reducing unnecessary calculations. Saldanha et al. widely collected sample data of depth map coding of different scenes and resolutions as input of machine learning models, and trained the model with the corresponding optimal coding mode as output label; in actual coding, the depth map data to be coded is input, and the model automatically outputs the most suitable coding mode to achieve fast and accurate prediction of mode selection. Chen et al. proposed a 3D-HEVC mode decision algorithm based on viewpoint texture correlation, which first analyzes the texture features of depth maps from different viewpoints in detail, then calculates the correlation of texture features between viewpoints and establishes a mathematical model, and scientifically evaluates the matching degree of different modes between different viewpoints based on this model, and selects the coding mode that can maintain texture continuity and consistency. Shen et al. used multiple correlations to select effective intra-frame modes in 3D-HEVC, comprehensively considered the temporal and spatial correlation of depth maps, the correlation between components and the correlation between views, analyzed the variation law of depth maps in the temporal and spatial dimensions, studied the intrinsic correlation between components, combined with the correspondence between views, and quantitatively integrated these correlations to build a model. Relying on this model, intra-frame modes that reflect image features and coding efficiency can be quickly and accurately selected during encoding. Song et al. proposed content-adaptive mode decision for low-complexity 3D-HEVC by analyzing factors such as depth map complexity and object distribution density. They classified the depth map content according to these factors, and deeply analyzed the coding characteristics and requirements for different types of regions to reduce unnecessary mode calculations. In the research on the improvement of 3D-HEVC depth map coding, the two main improvement methods of CU division and mode selection have some common defects: on the one hand, they are poor in adaptability, and some methods are highly dependent on specific data or scenes. Once they encounter depth map data that is significantly different from the training or design conditions, it is difficult to play an effective role and cannot guarantee the stable improvement of coding efficiency and quality. On the other hand, computational complexity has become a limiting factor. Some improved methods will introduce too much computation in the processing process, especially when facing high-resolution or complex scene depth maps, which seriously slows down the overall encoding speed. In addition, the generalization ability of the model is generally insufficient. Some machine learning-based methods require a large number of samples for training, which is not only costly, but also unable to accurately select the optimal strategy when facing new depth map scenes, resulting in large fluctuations in encoding efficiency.

[0005] In recent years, deep learning has shown great potential in the field of video coding. Thanks to its powerful feature learning and nonlinear modeling capabilities, it has brought new breakthroughs to video coding. Although existing deep learning-based video coding methods have made some progress, they also have some common problems. When processing depth maps, some models do not fully utilize the unique characteristics of depth maps (such as the distribution of depth values, the particularity of edges, etc.), resulting in a large space for improving coding efficiency and quality. Summary of the invention

[0006] In view of the technical problem of high complexity of existing 3D-HEVC depth map encoding methods, the present invention proposes a 3D-HEVC depth map fast encoding method based on the DCAE (Dual-Channel Attention Enhanced) model, which replaces the original CU division method in HTM, performs CU division according to the prediction results, effectively improves the encoding speed, and maintains the video quality while reducing the encoding complexity.

[0007] In order to achieve the above object, the technical solution of the present invention is implemented as follows: a 3D-HEVC depth map fast encoding method based on a DCAE model, the steps of which are as follows:

[0008] Step 1: Use 3D video sequences to construct a data set, encode the video sequences as the true labels of the video sequences, and form a training set;

[0009] Step 2: Calculate the CTU complexity parameter and normalized QP value of each coding tree unit;

[0010] Step 3: Construct a dual-channel attention enhancement model. The dual-channel attention enhancement model contains two channels. Each channel is equipped with a convolutional feature extraction module. The convolutional feature extraction modules of the two channels are connected to the attention mechanism module. The CTU complexity parameter and the normalized QP value are concatenated with the output features of the attention mechanism module and then input into the fully connected layer. Use the training set of step 1 to train the dual-channel attention enhancement model to obtain a trained DCAE model.

[0011] Step 4: Input the test sequence into the trained DCAE model to obtain the predicted probability of whether the CU is divided. According to the output predicted probability and decision method, determine whether the CU continues to be divided and complete the depth division process of the entire coding book unit.

[0012] Preferably, the video sequence in the data set is a coding tree unit of size 64×64, and the video sequence in the training set is completely separated from the test sequence by at least 50 frames; the method for encoding the video sequence is as follows: HTM16.0 performs CU division on the coding tree unit of the depth map according to its own coding rules, and for each coding tree unit, records whether it is further divided during the encoding process, if the CU continues to be divided, the label is set to 1, if it is no longer divided, the label is set to 0, and the division result is used as the real label; a 64×64 coding tree unit includes 1 64×64 CU, 4 32×32 CUs, and 16 32×32 CUs in the CU division process, and a total of 21 CUs need to be divided.

[0013] Preferably, the method for calculating the CTU complexity parameter of each coding tree unit is: for each coding tree unit, calculating its standard deviation and classifying it into different texture complexity levels according to a preset range, and classifying the coding tree unit into different quantization levels according to the range of QP values; determining the CTU complexity parameter in combination with the texture complexity level and the quantization level;

[0014] The normalized QP value is calculated by: corresponding to 52 quantization steps, multiplying the QP value by 1 / 51 to normalize it to between 0 and 1.

[0015] Preferably, the calculation method for determining the CTU complexity parameter is as follows: calculating the standard deviation of the depth map of the 64×64 coding tree unit, defining the texture complexity level of the coding tree unit as three levels of 0, 1, and 2 according to the ranges of 0 to 9, 10 to 19, and greater than 19, respectively; dividing the quantization level of the coding tree unit into three levels according to the range of the QP value, if the QP value is between 0 and 34, the level is 0, if the QP value is between 35 and 44, the level is 1, and if the QP value is between 45 and 51, the level is 2; combining the texture complexity level represented by the standard deviation with the quantization level of the QP value, calculating the CTU complexity parameter, and determining the complexity level of the CTU.

[0016] Preferably, the current coding tree unit texture complexity level is:

[0017]

[0018] Where n is the index of the current coding tree unit, σ n represents the standard deviation of the pixel values ​​of the current coding tree unit;

[0019] The quantization level of the current coding tree unit

[0020]

[0021] Among them, QP nis the quantization parameter of the current coding tree unit;

[0022] The CTU complexity parameter of the current coding tree unit is:

[0023]

[0024] Preferably, the depth map of the coding tree unit is transmitted to the convolutional feature extraction module after the average pooling operation is performed through one channel, and the depth map of the coding tree unit is transmitted to the convolutional feature extraction module after the mean value is removed through another channel. The convolutional feature extraction modules of the two channels each include three convolutional layers connected in sequence to perform multi-scale fusion of spatial information in the video; the feature map generated by the two convolutions after each convolutional layer is input into the attention mechanism module, and the generated feature map is compressed and stimulated in sequence to obtain the attention weight, and the attention weight is multiplied by the feature map output by the third convolutional layer to obtain the output feature; the output feature, the normalized QP value and the CTU complexity parameter are spliced ​​and input into the fully connected layer, and the prediction result is obtained through the output layer.

[0025] Preferably, when performing a compression operation, the input feature map of size H×W×C is converted into a 1×1×C vector through global average pooling; when performing an excitation operation, the operation is performed through two fully connected layers I: the first fully connected layer I compresses the 1×1×C vector, and the compressed vector is increased nonlinearly through a ReLU activation function; the second fully connected layer I restores the vector processed by the ReLU activation function to a 1×1×C vector, and limits the output value to between 0 and 1 through a sigmoid activation function to generate an attention weight;

[0026] The fully connected layer undergoes three full connections. After the first full connection, the normalized QP value and CTU complexity parameters are introduced and combined with the output features, and then the next full connection operation is performed; the prediction result is compared with the set threshold to determine the final depth of the CU.

[0027] Preferably, the dual-channel attention enhancement model uses binary cross entropy as the loss function, and adjusts the model parameters according to the loss function value through the back propagation algorithm to make the model prediction result closer to the true label, thereby optimizing the accuracy of the model prediction of the CU partition.

[0028] Preferably, the de-averaging retains the size of the input depth map to 64×64, and the average pooling size is 4×4 to reduce the depth map to 16×16;

[0029] The convolution kernel sizes of the three convolution layers are set to 4×4, 2×2, and 2×2, respectively, and the ReLU activation function is applied after each convolution layer; for the 16 convolution kernels in the first convolution layer, the sizes of the feature maps generated by the two channels are 16×16 and 4×4, respectively; for the 24 convolution kernels in the second convolution layer, the sizes of the feature maps generated by the two channels are 8×8 and 2×2, respectively; for the 32 convolution kernels in the third convolution layer, the sizes of the feature maps generated by the two channels are 4×4 and 1×1, respectively;

[0030] 50% of the first fully connected features are randomly dropped.

[0031] When the prediction result is greater than the set threshold, CU division continues, otherwise the division stops.

[0032] Preferably, the trained model replaces the original depth map CU division method in HTM, and the trained DCAE model processes the depth map of the coding tree unit in the input test sequence to obtain the predicted division probability of each CU and predict the CU division method; the CU division method is determined according to the predicted division probability and the decision mechanism. After the CU division is completed, the encoder continues with subsequent processing, performs further encoding operations on the divided CU, outputs the generated bit stream, and completes the entire encoding process.

[0033] Compared with the prior art, the invention has the following beneficial effects: by deeply analyzing the encoding characteristics of the depth map, designing a dual-channel feature transmission structure, introducing an attention mechanism after convolution to enhance the key feature learning ability, automatically identifying and emphasizing important feature channels through compression operations and excitation operations, effectively utilizing the characteristics of the depth map and the advantages of the model architecture, and improving the adaptability to different video contents; and combining the standard deviation with the quantization parameter (QP) to characterize the CTU complexity, providing more accurate information for encoding decisions, so as to achieve fast CU division decisions, effectively reduce encoding complexity, and improve encoding speed. Experimental results show that compared with HTM16.0, the DCAE model-based method proposed in the invention can save an average of 48.08% of encoding time while maintaining video quality, and the average BDBR increases by only 0.31%. It significantly improves the encoding efficiency while maintaining video quality, and effectively reduces the encoding complexity, providing strong support for the development of 3D video encoding technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 It is a schematic diagram of CTU depth level division of the present invention.

[0036] Figure 2 It is a flow chart of the present invention.

[0037] Figure 3 This is a schematic diagram of the DCAE model of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Example 1

[0040] like Figure 2 As shown, a 3D-HEVC depth map fast encoding method based on the DCAE model is used to accurately predict whether the CU is divided according to the relevant features of the CU in order to solve the problem of high complexity in 3D-HEVC depth map encoding. The specific steps are as follows:

[0041] Step 1: Use the 3D standard video test sequence to build a data set, and use HTM16.0 to encode the video sequence in the data set as the division result of the video sequence.

[0042] The dataset is constructed using the official 3D standard video test sequence of JCT-3V. In order to maximize the number and types of datasets, a large number of videos are selected. In order to avoid overlap between training data and test data, the training frames and test frames are completely separated by at least 50 frames.

[0043] When collecting the division results and setting the labels, the video sequence is encoded with the help of HTM16.0. During the encoding process, HTM16.0 will divide the coding tree unit (CTU) of the depth map into CUs according to its own encoding rules. For each CTU, it is recorded whether it is further divided during the encoding process. If the CU continues to be divided, the label is set to 1, and if it is no longer divided, the label is set to 0. This information constitutes the division result.

[0044] like Figure 1As shown in the figure, the way of dividing the depth level of CTU is as follows: Initially, CTU is a 64×64 CU with a depth level of CU_Depth=0. The 64×64 CU can be divided into 4 32×32 CUs, and the depth level of these 4 CUs becomes CU_Depth=1. The 32×32 CU can be further divided into 4 16×16 CUs, and the depth level of these CUs becomes CU_Depth=2. The 16×16 CU can be further divided into 8×8 CUs, and its depth level becomes CU_Depth=3. Therefore, a 64×64 CTU includes 1 64×64 CU, 4 32×32 CUs, and 16 32×32 CUs, and a total of 21 CUs need to be divided. Therefore, it is necessary to predict the division probability of 21 CUs to determine whether each CU continues to be divided.

[0045] Step 2: Calculate the CTU complexity parameter and normalized QP value of each CTU.

[0046] For each CTU, its standard deviation is calculated and classified into different texture complexity levels according to the range. At the same time, it is classified into different quantization levels according to the range of QP values. Then the two are combined to determine the CTU complexity parameter C and used as an external feature introduced in the fully connected layer of the DCAE model. The HEVC standard specifies 52 quantization steps, corresponding to 52 QPs (0 to 51). The QP value is normalized to between 0 and 1 by multiplying it with 1 / 51 and is introduced in the fully connected layer.

[0047] The calculation method of CTU complexity parameter C is as follows: calculate the standard deviation of the 64×64CTU depth map, and define the texture complexity level of CTU as 0, 1, and 2 according to the range of 0 to 9, 10 to 19, and greater than 19 respectively; according to the range of QP value, the quantization degree of CTU is divided into three levels. If QP is between 0 and 34, the level is 0, the level is 1 between 35 and 44, and the level is 2 between 45 and 51. The texture complexity represented by the standard deviation is combined with the quantization degree represented by QP to define the CTU complexity parameter C, so as to determine the complexity level of CTU, and then select the appropriate encoding strategy according to the complexity level.

[0048] In the depth map encoding process of 3D-HEVC, accurately measuring the texture complexity of the depth map is crucial to optimizing the encoding process. The present invention introduces the standard deviation to measure the texture complexity of the depth map, which is based on the characteristic that the standard deviation can be used as a statistic to measure the degree of data discreteness. For a CTU (64×64 pixels) depth map, its pixel value represents the distance information from the object to the camera, and usually has a certain range of values. By analyzing the distribution law of pixel values ​​in different situations, it is found that the standard deviation can effectively reflect the texture complexity of the depth map. First, the average value of the depth value in the depth map is calculated as follows:

[0049]

[0050] The standard deviation is then calculated as follows:

[0051]

[0052] in, is the average value of depth, x i is the depth value of the i-th pixel in the CTU, i = 1, 2, 3, ..., n, n is the number of pixels, and σ is the standard deviation of the calculated depth value in the block. This simple and effective calculation method can evaluate the texture complexity of the depth map in real time. According to the calculated standard deviation, three ranges are set through experiments to represent different depth map texture complexity levels, which are defined as follows:

[0053]

[0054] Here, S n Represents the current CTU texture complexity level, and n is the CTU index currently being processed. n Represents the standard deviation of the current CTU pixel value.

[0055] Quantization parameter (QP) also plays a key role in video encoding, reflecting the degree of quantization of video encoding. According to the range of QP, the degree of quantization is divided into three categories and recorded as quantization degree parameters, as follows:

[0056]

[0057] Here, Q n Represents the quantization level of the current CTU. n It is the quantization parameter of the current CTU.

[0058] In order to better comprehensively consider the impact of texture complexity and quantization parameters on encoding, the CTU complexity parameter C is defined, and the calculation method is:

[0059]

[0060] Among them, C n Represents the complexity of the current CTU. By setting different threshold intervals, the complexity level of the current CTU is determined according to the interval in which the combination of texture complexity and quantization degree falls. When the CTU complexity parameter C value is small, a simpler encoding strategy can be adopted. When the CTU complexity parameter C value is large, a more complex encoding strategy is required, such as using a smaller encoding block to ensure encoding quality.

[0061] Step 3: Construct a dual-channel attention enhancement (DCAE) model. The DCAE model contains two channels, each channel has three convolutional layers, and the convolutional layers of the two channels are connected to the attention mechanism module, which is connected to the fully connected layer through the connection layer. Use the data set in step 1 to train the DCAE model and obtain the trained DCAE model.

[0062] The output of the DCAE model is a vector of size 21, corresponding to the partition probability of different regions (CU).

[0063] like Figure 3 As shown, the present invention proposes to construct a DCAE model of a specific architecture for the prediction of CU partitioning. In the input stage of the model, the 64×64 input data is processed, the first channel of the DCAE model performs de-averaging to retain the original size of the input CU 64×64, and the second channel performs an average pooling operation on the input CU to reduce the size to 16×16, highlighting the background features and making the extracted features more diverse. The size of the average pooling is 4×4, which can better capture the local features of the image, avoid losing too much detail information due to the large pooling size, and will not effectively reduce the data dimension due to the small pooling size. In terms of extracting background features, this size of pooling operation can properly fuse and average the pixel values ​​of the background area, highlight the overall characteristics of the background, make the extracted features more diverse, and help the model better learn the feature differences between different areas in the depth map, and improve the encoding effect of the depth map.

[0064] In the convolution part, the two channels are convolved three times through the convolution layer respectively, and the convolution feature extraction module is used to perform multi-scale fusion of spatial information in the video. The convolution kernel sizes of the three convolution layers are set to 4×4, 2×2 and 2×2 respectively, and the ReLU activation function is applied after each convolution layer. Because depth map encoding is a complex process, there are nonlinear relationships between data. The ReLU activation function can help the model better fit these complex relationships, so that the model can learn richer feature representations. Secondly, the ReLU activation function is simple to calculate and can effectively improve the training speed of the model. For the 16 convolution kernels in the first convolution layer, the feature map sizes generated by the two channels are 16×16 and 4×4 respectively. For the 24 convolution kernels in the second convolution layer, the feature map sizes generated by the two channels are 8×8 and 2×2 respectively. For the 32 convolution kernels in the third convolution layer, the feature map sizes generated by the two channels are 4×4 and 1×1 respectively.

[0065] The feature maps generated by the last two convolutions of each layer are input into the attention mechanism module, and the generated feature maps are compressed and stimulated respectively to obtain the attention weights. The attention weights are multiplied with the feature maps output by the original convolution layer to enhance the influence of important feature channels and suppress the influence of unimportant channels. When performing the compression operation, the attention mechanism module first converts the input H×W×C feature map into a 1×1×C vector through global average pooling. When performing the stimulation operation, two fully connected layers are operated. The first fully connected layer compresses the 1×1×C vector to a smaller dimension (here the compression ratio is 2), and the compressed vector is increased nonlinearly through a ReLU activation function. Then, the second fully connected layer restores it to the original dimension 1×1×C, and finally a sigmoid activation function is used to limit the output value between 0 and 1 to generate the attention weight.

[0066] The features are concatenated and input into the fully connected layer. There are three full connections in total. The three fully connected layers are mainly set to gradually perform more complex nonlinear transformations and information fusion on the input features to improve the model's expressiveness and prediction accuracy. After the first full connection, the normalized QP value and the previously defined complexity parameter C are introduced and combined with the features, followed by the next full connection operation to enable further fusion with the features. At the same time, in order to avoid overfitting, 50% of the features of the first fully connected layer will be randomly discarded. Finally, the prediction result of the model is obtained through the output layer, and the output size is 21, which realizes the prediction of CU division; it is compared with the set threshold of 0.5 to determine the final depth of CU. When the prediction result is greater than the set threshold of 0.5, the division continues, otherwise the division stops.

[0067] The DCAE model uses binary cross entropy as the loss function. The model prediction result is compared with the true label, which indicates whether the CU should continue to be divided. When calculating the loss function, for each sample in the dataset, the loss value is calculated based on its true label and the model prediction probability, and then the loss values ​​of all samples are averaged to obtain the final loss function value. Through the back-propagation algorithm, the model parameters are adjusted according to the loss function value to make the model prediction result closer to the true label, thereby optimizing the accuracy of the model prediction of CU partitioning.

[0068] Step 4: Integrate the trained DCAE model into HTM16.0, replace the original CU division method in the depth map, predict whether the CU in the test sequence is divided, and determine whether the CU continues to be divided and complete the depth division process of the entire CTU based on the prediction probability and decision-making method output by the DCAE model.

[0069] like Figure 2 As shown in the figure, in actual application, the trained model replaces the original depth map CU division method in HTM, and the overall process is as follows: after starting encoding, input the video sequence, and process viewpoints 1, 2, and 3 in the multi-view video. When processing the depth map, use the trained DCAE model to operate the depth map, that is, call the trained DCAE model to process the CU, obtain the predicted division probability of each CU, and predict the CU division method. The CU division method is determined according to the prediction results of the DCAE model and the decision-making mechanism. After completing the CU division, the encoder continues with subsequent processing and performs further encoding operations on the divided CU. After the above processing, the generated bit stream is output, the encoding is ended, and the entire encoding process is completed.

[0070] In order to evaluate the effectiveness of the 3D-HEVC depth map fast encoding method proposed in the present invention, the designed model is integrated into the reference software HTM 16.0, and then tested using the official 3D standard video test sequence of JCT-3V, a total of 8 video sequences Kendo, Balloons, Newspaper, GT_Fly, Shark, Poznan_Hall2, Poznan_Street and Undo_Dancer, including 1024×768 and 1920×1088 resolutions. The present invention uses the All Intra (AI) mode to test the performance of 3D-HEVC intra-frame coding, and uses the saved coding time percentage TS and the bit rate increment ΔBDBR as the measurement criteria. The calculation formulas for the saved coding time percentage TS and the bit rate increment ΔBDBR are:

[0071]

[0072]

[0073] Among them, TS is the percentage of encoding time saved, T pro represents the encoding time of the algorithm proposed by the present invention, T HTM represents the encoding time of the reference model HTM 16.0. In addition, the bit rate increment BDBR ( Delta bitrate) is used as a criterion for evaluating the coding efficiency gains achieved by various methods while maintaining a consistent target quality.

[0074] Table 1 shows the overall experimental results of the method of the present invention. It can be clearly seen from Table 1 that compared with the reference model HTM16.0, the dual-channel attention enhancement model proposed in the present invention can greatly shorten the 3D-HEVC depth map encoding time while maintaining excellent performance on the 8 test sequences provided. The experimental results show that the encoding time of the present invention is reduced by an average of 48.08%, and the average increase in bit rate increment BDBR is 0.31%.

[0075] Table 1. Comparison of test performance between the present invention and HTM16.0

[0076]

[0077] In order to comprehensively evaluate the method of the present invention, two representative sequences were selected from the video sequences of 1024×768 and 1920×1088 resolutions, and the video synthesis views before and after encoding were compared. Table 2 shows the subjective quality of the method proposed in the present invention and the original encoder HTM-16.0 in terms of synthetic views. It can be seen from Table 2 that although the objective data has slightly decreased, the subjective quality has not decreased, which further confirms that the method proposed in the present invention has the ability to reduce the complexity of intra-frame coding of 3D-HEVC depth maps.

[0078] Table 2. Comparison of subjective quality of synthetic views

[0079]

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A 3D-HEVC depth map fast encoding method based on DCAE model, characterized in that: The steps are as follows: Step 1: Use 3D video sequences to construct a data set, encode the video sequences as the true labels of the video sequences, and form a training set; Step 2: Calculate the CTU complexity parameter and normalized QP value of each coding tree unit; Step 3: Construct a dual-channel attention enhancement model. The dual-channel attention enhancement model contains two channels. Each channel is equipped with a convolutional feature extraction module. The convolutional feature extraction modules of the two channels are connected to the attention mechanism module. The CTU complexity parameter and the normalized QP value are concatenated with the output features of the attention mechanism module and then input into the fully connected layer. Use the training set of step 1 to train the dual-channel attention enhancement model to obtain a trained DCAE model. Step 4: Input the test sequence into the trained DCAE model to obtain the predicted probability of whether the CU is divided. According to the output predicted probability and decision method, determine whether the CU continues to be divided and complete the depth division process of the entire coding book unit.

2. The 3D-HEVC depth map fast encoding method based on the DCAE model according to claim 1, characterized in that: The video sequence in the data set is a coding tree unit of size 64×64, and the video sequence in the training set is completely separated from the test sequence by at least 50 frames; the method for encoding the video sequence is as follows: HTM16.0 divides the coding tree unit of the depth map into CUs according to its own coding rules, and for each coding tree unit, records whether it is further divided during the encoding process. If the CU continues to be divided, the label is set to 1, and if it is no longer divided, the label is set to 0, and the division result is used as the real label; a 64×64 coding tree unit includes 1 64×64 CU, 4 32×32 CUs, and 16 32×32 CUs in the CU division process, and a total of 21 CUs need to be divided.

3. The 3D-HEVC depth map fast encoding method based on the DCAE model according to claim 1 or 2, characterized in that: The method for calculating the CTU complexity parameter of each coding tree unit is as follows: for each coding tree unit, calculating its standard deviation and classifying it into different texture complexity levels according to a preset range, classifying the coding tree unit into different quantization levels according to the range of QP values; and determining the CTU complexity parameter in combination with the texture complexity level and the quantization level; The normalized QP value is calculated by: corresponding to 52 quantization steps, multiplying the QP value by 1 / 51 to normalize it to between 0 and 1.

4. The 3D-HEVC depth map fast encoding method based on the DCAE model according to claim 3, characterized in that: The calculation method for determining the CTU complexity parameter is as follows: calculate the standard deviation of the depth map of the 64×64 coding tree unit, and define the texture complexity level of the coding tree unit as 0, 1, and 2 according to the ranges of 0 to 9, 10 to 19, and greater than 19, respectively; divide the quantization level of the coding tree unit into three levels according to the range of the QP value, if the QP value is between 0 and 34, the level is 0, if the QP value is between 35 and 44, the level is 1, and if the QP value is between 45 and 51, the level is 2; combine the texture complexity level represented by the standard deviation with the quantization level of the QP value, calculate the CTU complexity parameter, and determine the complexity level of the CTU.

5. The 3D-HEVC depth map fast encoding method based on the DCAE model according to claim 4, characterized in that: The current coding tree unit texture complexity level is: Where n is the index of the current coding tree unit, σ n represents the standard deviation of the pixel values ​​of the current coding tree unit; The quantization level of the current coding tree unit Among them, QP n is the quantization parameter of the current coding tree unit; The CTU complexity parameter of the current coding tree unit is:

6. The 3D-HEVC depth map fast encoding method based on the DCAE model according to claim 4 or 5, characterized in that: The depth map of the coding tree unit is transmitted to the convolutional feature extraction module after average pooling operation through one channel, and the depth map of the coding tree unit is transmitted to the convolutional feature extraction module after de-averaging through another channel. The convolutional feature extraction modules of the two channels include three convolutional layers connected in sequence to perform multi-scale fusion of spatial information in the video; the feature map generated by the two convolutions after each convolutional layer is input into the attention mechanism module, and the generated feature map is compressed and stimulated in sequence to obtain the attention weight, and the attention weight is multiplied by the feature map output by the third convolutional layer to obtain the output feature; the output feature, normalized QP value and CTU complexity parameter are spliced ​​and input into the fully connected layer, and the prediction result is obtained through the output layer.

7. The 3D-HEVC depth map fast encoding method based on the DCAE model according to claim 6, characterized in that: When performing the compression operation, the input feature map of size H×W×C is converted into a 1×1×C vector through global average pooling; when performing the excitation operation, it is operated through two fully connected layers I: the first fully connected layer I compresses the 1×1×C vector, and the compressed vector is added with nonlinearity through a ReLU activation function; the second fully connected layer I restores the vector processed by the ReLU activation function to a 1×1×C vector, and limits the output value to between 0 and 1 through a sigmoid activation function to generate attention weights; The fully connected layer undergoes three full connections. After the first full connection, the normalized QP value and CTU complexity parameters are introduced and combined with the output features, and then the next full connection operation is performed; the prediction result is compared with the set threshold to determine the final depth of the CU.

8. The 3D-HEVC depth map fast encoding method based on the DCAE model according to claim 7, characterized in that: The dual-channel attention enhancement model uses binary cross entropy as the loss function, and adjusts the model parameters according to the loss function value through the back propagation algorithm to make the model prediction result closer to the true label, thereby optimizing the accuracy of the model prediction CU partition.

9. The 3D-HEVC depth map fast encoding method based on the DCAE model according to claim 7 or 8, characterized in that: The de-averaging retains the size of the input depth map at 64×64, and the average pooling size is 4×4 to reduce the depth map to 16×16; The convolution kernel sizes of the three convolution layers are set to 4×4, 2×2, and 2×2, respectively, and the ReLU activation function is applied after each convolution layer; for the 16 convolution kernels in the first convolution layer, the sizes of the feature maps generated by the two channels are 16×16 and 4×4, respectively; for the 24 convolution kernels in the second convolution layer, the sizes of the feature maps generated by the two channels are 8×8 and 2×2, respectively; for the 32 convolution kernels in the third convolution layer, the sizes of the feature maps generated by the two channels are 4×4 and 1×1, respectively; 50% of the first fully connected features are randomly discarded; When the prediction result is greater than the set threshold, CU division continues, otherwise the division stops.

10. The 3D-HEVC depth map fast encoding method based on the DCAE model according to claim 7 or 8, characterized in that: The trained model replaces the original depth map CU partitioning method in HTM. The trained DCAE model processes the depth map of the coding tree unit in the input test sequence to obtain the predicted partitioning probability of each CU and predict the CU partitioning method. The CU partitioning method is determined according to the predicted partitioning probability and the decision-making mechanism. After the CU partitioning is completed, the encoder continues with subsequent processing, performs further encoding operations on the divided CU, outputs the generated bit stream, and completes the entire encoding process.

Images