Low-power-consumption video transcoding method based on deep learning

Abstract

The invention discloses a low-power-consumption video transcoding method based on deep learning, and the method comprises the steps: carrying out the transcoding from a DVC coding video to an HEVC coding video, marking an initial feature vector according to the CU division depth and the PU division mode in the transcoding process, and obtaining a data set; constructing a lightweight deep learning model and training according to the data set to obtain a division depth prediction model; the feature vectors of the DVC video frames are processed and then input into a division depth prediction model, and optimal prediction choices of the CU division depth and the PU division mode are obtained; hEVC coding is carried out on the DVC video frame according to the prediction optimal selection of the CU division depth and the PU division mode; according to the method, the space-time correlation of the initial features is fully mined in global and local levels, and the prediction accuracy of the CU division depth and the PU division mode is improved, so that the transcoding compression performance is improved while the speed of transcoding from DVC to HEVC is increased.

Description

Technical Field

[0001] The present application relates to the field of video transcoding, and in particular to a low-power video transcoding method based on deep learning. Background Art

[0002] As people's demand for high-quality production and living grows, IoT video communication applications such as video surveillance and video interaction are developing rapidly, which has put tremendous pressure on network resources such as energy, bandwidth, and computing power. Therefore, low-power video communication is crucial in the IoT environment, especially for resource-constrained terminal devices. Against this background, this application studies low-power encoding technology suitable for resource-constrained terminals, aiming to extend video communication time and improve video compression performance under the same conditions.

[0003] Traditional video coding standards integrate prediction and transformation techniques at the encoder end to deeply mine source correlations, achieving efficient video compression performance at the expense of high coding complexity. Distributed video coding (DVC), on the other hand, offloads the source correlation mining task to the decoder end, freeing the encoder from heavy computational overhead. Therefore, lightweight encoding is DVC's advantage over traditional video coding, but at the expense of relatively high decoding complexity. Due to the characteristics of both types of video coding, running either coding technique alone on resource-constrained end devices cannot guarantee low-power video coding. Therefore, transcoding from DVC to traditional video coding (DVC transcoding), which combines the advantages of both, has become a viable solution for low-power video coding. In the DVC transcoding architecture, the computationally more complex DVC decoding and traditional video encoding processes are offloaded to servers with greater computing power and a sustainable energy supply; while the less computationally complex DVC encoding and traditional video decoding processes are handled by the end device. DVC transcoding offers two main advantages: First, its distributed-to-centralized architecture is naturally well-suited for IoT applications. Second, modules with high computational complexity are all centralized on the server for processing, which facilitates centralized optimization of transcoding speed. This reduces the computational burden on the transcoding server, enabling fast DVC transcoding. This is crucial for achieving low-power video communication within the DVC transcoding framework.

[0004] In terms of transcoding efficiency, the integration of machine learning and DVC coding technology has achieved fruitful results, enabling more efficient video transcoding. Existing research has employed Fisher's discriminant, Naive Bayes, and Support Vector Machines (SVM) to predict the depth and prediction unit (PU) of coding units (CUs) in High Efficiency Video Coding (HEVC), effectively reducing HEVC coding complexity. These algorithms significantly improve DVC transcoding speed at the expense of acceptable compression performance. Building on this approach, subsequent researchers employed the SVM recursive feature elimination algorithm to filter key features from a heuristically selected feature set, further optimizing transcoding efficiency. However, these methods primarily rely on online learning mechanisms, making it difficult to provide extensive and diverse training data for the prediction model to extract effective features. Furthermore, traditional machine learning methods have inherent limitations in feature learning. These factors, combined with the inherent limitations of traditional machine learning methods in feature learning, have hindered improvements in CU depth prediction accuracy, ultimately hindering the coordinated optimization of compression performance and transcoding speed. To overcome these limitations, relevant research has begun to introduce deep learning networks to accelerate DVC transcoding. Although this research provides a richer training data set, the correlation between its input features is weak and there is a lack of effective fusion in neural network processing. Therefore, it is not conducive to the effective mining and full utilization of features, and the transcoding efficiency needs to be improved. Summary of the Invention

[0005] In response to the above-mentioned deficiencies in the prior art, the present invention provides a low-power video transcoding method based on deep learning, which solves the problem that the existing video transcoding method relies on online learning mechanisms, is difficult to provide broader and more diverse training data for the prediction model to extract effective features, has weak correlation between features and lacks effective fusion in neural network processing, is not conducive to the effective mining and full utilization of features, and leads to poor coordinated optimization of compression performance and transcoding speed.

[0006] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a low-power video transcoding method based on deep learning for fast transcoding from DVC encoding to HEVC encoding, comprising:

[0007] S1. Data preparation: Transcode DVC-encoded video to HEVC-encoded video, and mark the initial feature vector according to the CU division depth and PU division mode during the transcoding process to obtain a dataset;

[0008] S2. Model construction: Build a lightweight deep learning model and train it based on the dataset to obtain a partitioning depth prediction model;

[0009] S3, partition prediction: the DVC video frame feature vector is processed and input into the partition depth prediction model to obtain the best prediction selection of CU partition depth and PU partition mode;

[0010] S4. Video coding: HEVC encoding is performed on the DVC video frame according to the predicted best selection of CU partition depth and PU partition mode.

[0011] Furthermore: S1 includes:

[0012] S11, decoding the DVC encoded video to obtain a reconstructed video frame and an initial feature vector;

[0013] S12, performing HEVC transcoding on the reconstructed video frame to obtain real data of the CU and PU partitioning modes;

[0014] S13. Label the initial feature vector according to the real data of the CU and PU partitioning patterns to obtain a data set.

[0015] Furthermore: S3 includes:

[0016] S31, processing the feature vector of the DVC video frame to be transcoded and inputting it into the partition depth prediction model to obtain the prediction class label of the CU partition depth;

[0017] S32: Predict the PU partition mode according to the prediction class label of the CU partition depth.

[0018] Further: S31 includes:

[0019] S311, extracting a luminance image containing intra-frame correlation and a residual image containing inter-frame correlation from a feature vector of a DVC video frame to be transcoded, and inputting the images into a partitioned depth prediction model;

[0020] S312, respectively performing compression and excitation on the luminance image containing intra-frame correlation and the residual image containing inter-frame correlation to extract a low-dimensional feature map;

[0021] S313: Perform convolution processing on the obtained low-dimensional feature map after feature fusion to obtain a higher-dimensional feature space and obtain the predicted class label of the CU partition depth.

[0022] Furthermore, the feature fusion process performed in S313 is expressed as follows:

[0023]

[0024] in, represents the feature fusion result on the cth channel, c∈{1,2,...,C} represents the cth channel, C represents the number of output channels, u c and zc Respectively represent the values of the low-dimensional feature map corresponding to the brightness image before feature fusion and the low-dimensional feature map corresponding to the residual image on the cth channel, α c and β c are all weight matrices.

[0025] Further: α c and β c The value at position (i, j) is represented as:

[0026]

[0027] in, and They are control parameters and The value at position (i,j), and They are λ α and λ β Parameter matrix on the cth channel, controlling parameter λ α and λ β The expression is:

[0028] λ a =W2δ(W1U)

[0029] λ β =W2δ(W1Z)

[0030] Among them, δ() represents the ReLU function, W1 and W2 are the weights of the 1×1 convolutional layer, and U and Z represent the output of the changes to the brightness image and residual image before feature fusion, respectively.

[0031] Furthermore: the prediction class labels of the CU partition depth include needing further partitioning and not needing further partitioning. For the CU that does not need further partitioning, PU partition mode prediction is performed.

[0032] Further: in S32, the specific process of performing PU partition mode prediction according to the prediction class label of the CU partition depth is: starting from the top depth of the quadtree coding, PU partition mode prediction is performed on CUs that do not need further partitioning from top to bottom at each depth level.

[0033] The beneficial effects of the present invention are:

[0034] 1. Minimize HEVC encoding complexity by exploring the correlation between DVC and HEVC, thereby accelerating transcoding;

[0035] 2. Using top-down CU partition depth and PU partition mode prediction helps to realize partition selection earlier, thereby reducing the complexity of quadtree coding;

[0036] 3. Selecting the luminance image containing intra-frame correlation and the residual image containing inter-frame correlation in the DVC decoding information as two inputs solves the problem of weak inter-feature correlation and lack of effective fusion in neural network processing, which is not conducive to effective feature mining and full utilization, resulting in low transcoding efficiency;

[0037] 4. A lightweight deep learning model is used to seamlessly integrate the partitioned depth prediction model into HEVC quadtree coding with minimal additional computation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of a low-power video transcoding method based on deep learning;

[0039] Figure 2 A schematic diagram of a low-power video transcoding method based on deep learning;

[0040] Figure 3 An example diagram of a partitioned depth prediction model. DETAILED DESCRIPTION

[0041] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0042] like Figure 1 As shown, a low-power video transcoding method based on deep learning is used for fast DVC to HEVC transcoding, including:

[0043] S1. Data preparation: Transcode DVC-encoded video to HEVC-encoded video, and mark the initial feature vector according to the CU division depth and PU division mode during the transcoding process to obtain a dataset;

[0044] S2. Model construction: Build a lightweight deep learning model and train it based on the dataset to obtain a partitioning depth prediction model;

[0045] S3, partition prediction: the DVC video frame feature vector is processed and input into the partition depth prediction model to obtain the best prediction selection of CU partition depth and PU partition mode;

[0046] S4. Video coding: HEVC encoding is performed on the DVC video frame according to the predicted best selection of CU partition depth and PU partition mode.

[0047] Preferably, the data set can be divided into a training data set, a validation data set and a test data set.

[0048] The basic idea of fast transcoding in this application is to minimize the HEVC encoding complexity by mining the correlation between DVC and HEVC, thereby achieving accelerated transcoding; in HEVC, the recursive rate-distortion optimization (RDO) process in quadtree coding is the main reason for the high encoding complexity. If the "best choice" of CU partition depth and PU partition mode can be known in advance, redundant RDO calculation steps can be effectively avoided. In this context, the selection of CU partition depth and PU partition can be regarded as two classification problems respectively-let Y represent the label set of the selected category, and X is the feature set related to Y (in this application, X is the initial feature vector). In essence, the partition depth prediction model learns the probability distribution P(Y|X) or the decision function Y=f(X) through the training data set, and then predicts whether the current CU should continue to be divided or which PU partition mode should be selected based on the input features.

[0049] The training of the partition depth prediction model is completed offline. A schematic diagram of the low-power video transcoding method based on deep learning proposed in this embodiment is shown in FIG. Figure 2 As shown, specifically, S1 includes:

[0050] S11, decoding the DVC encoded video to obtain a reconstructed video frame and an initial feature vector;

[0051] S12, performing HEVC encoding on the reconstructed video frame to obtain real data of CU partition depth and PU partition mode;

[0052] S13. Label the initial feature vector according to the real data of the CU partition depth and the PU partition mode to obtain a data set.

[0053] Furthermore, in this embodiment, S3 includes:

[0054] S31, processing the feature vector of the DVC video frame to be transcoded and inputting it into the partition depth prediction model to obtain the prediction class label of the CU partition depth;

[0055] S32: Predict the PU partition mode according to the prediction class label of the CU partition depth.

[0056] Specifically, in this embodiment, the classification accuracy of the partitioned depth prediction model has an important impact on the performance of the proposed fast DVC transcoding. The basic idea of the design of the partitioned depth prediction model is to seamlessly integrate the partitioned depth prediction model into the quadtree coding of HEVC at the cost of as little additional computation as possible. For HEVC quadtree coding, the maximum size of the coding tree unit is 64×64 pixels. As the CU partition depth increases, the size of the CU gradually decreases. This means that as the CU depth is divided, the image information provided by the CU will gradually decrease and become more localized, and therefore is not enough to support the use of deep networks. In addition, more network layers will require more execution time. In order to achieve a trade-off between network processing latency and classification accuracy, we focus on existing lightweight network models because their architectures are more mature and modifications to them can achieve twice the result with half the effort.

[0057] Preferably, MobileNetV3 is finally selected as the reference network for the depth prediction model. This is because MobileNetV3 has a relatively high classification accuracy and a fast processing speed. This performance is mainly attributed to the bottleneck module in MobileNetV3. In addition, the bottleneck module in MobileNetV3 is reused many times. In this case, we can easily trim the number of times the bottleneck module is used according to actual needs. Through exhaustive testing, the lightweight deep learning network proposed in this application, such as Figure 3 shown.

[0058] exist Figure 3 In the example, “conv2d,3×3” indicates a 2D convolution with a kernel size of 3×3. “bneck,3×3” indicates the bottleneck module in MobileNetV3. “Feature fusion” refers to the feature fusion method proposed in this paper. “conv2d,1×1” indicates a 2D convolution with a kernel size of 1×1. “NBN” indicates that batch normalization is not required. “pool,(m / 4)×(m / 4)” indicates an average pooling operation, whose window size depends on the size of the input image. The output is y i ∈{0, 1, ..., K}, d∈{0, 1, 2} represents the class label y of the CU partition depth at the dth depth i The output of K represents the total number of class labels.

[0059] Furthermore, in this embodiment, S31 includes:

[0060] S311, extracting a luminance image containing intra-frame correlation and a residual image containing inter-frame correlation from a feature vector of a DVC video frame to be transcoded, and inputting the images into a partitioned depth prediction model;

[0061] S312, respectively performing compression and excitation on the luminance image containing intra-frame correlation and the residual image containing inter-frame correlation to extract a low-dimensional feature map;

[0062] S313: Perform convolution processing on the obtained low-dimensional feature map after feature fusion to obtain a higher-dimensional feature space and obtain the predicted class label of the CU partition depth.

[0063] Preferably, taking into account the essence of HEVC quadtree coding, which is to obtain the best coding mode by utilizing the inter-frame and intra-frame correlations contained in the video image. Through testing, the luminance image containing rich intra-frame correlations and the residual image containing significant inter-frame correlations in the DVC decoding information are selected as two inputs. Specifically, for a CU with an m×m pixel size, the luminance and residual information of the image within the CU size are input into the proposed network respectively; subsequently, a 3×3 two-dimensional convolution and a MobileNetV3 bottleneck layer with squeeze and excitation functions are applied to extract low-dimensional feature maps, and then the low-dimensional feature maps are fused in the feature fusion layer; after that, they are sequentially processed by a MobileNetV3 bottleneck layer without squeeze and excitation functions and a variety of variant 1x1 convolution layers to obtain a higher-dimensional feature space; finally, the classification results are output.

[0064] Specifically, in this embodiment, the feature fusion process performed in S313 is expressed as follows:

[0065]

[0066] in, represents the feature fusion result on the cth channel, c∈{1,2,...,C} represents the cth channel, C represents the number of output channels, u c and z c Respectively represent the values of the low-dimensional feature map corresponding to the brightness image before feature fusion and the low-dimensional feature map corresponding to the residual image on the cth channel, α c and β c Both are weight matrices, which help the network pay attention to changes in brightness and texture.

[0067] α c and β c The value at position (i, j) is represented as:

[0068]

[0069] in, and They are control parameters and The value at position (i,j), and They are λα and λ β Parameter matrix on the cth channel, controlling parameter λ α and λ β The expression is:

[0070] λ a =W2δ(W1U)

[0071] λ β =W2δ(W1Z)

[0072] Among them, δ() represents the ReLU function, W1 and W2 are the weights of the 1×1 convolutional layer, and U and Z represent the output of the changes to the brightness image and residual image before feature fusion, respectively.

[0073] Specifically, the predicted class labels for the CU split depth include those indicating further splitting is needed and those indicating no further splitting is needed. For CUs that do not require further splitting, the PU split mode is predicted. For the CU split depth, the selected class label set Y has only two classes {0, 1}. 0 indicates that the CU does not need to be split, and 1 indicates that further splitting is required. However, the prediction of the PU split mode is slightly different. This is because for a CU using inter-frame transcoding, there are a total of eight PU split modes to choose from, but the frequency of use of each mode is uneven.

[0074] This embodiment uses a partitioning pattern prediction model with the same structure as the partitioning depth prediction model but different parameters. The partitioning pattern prediction model is trained to predict the three most commonly used PU partitioning patterns at the dth layer depth. Accordingly, the classification label Y contains four categories {0, 1, 2, 3}—where 0, 1, and 2 represent the three most frequently used PU partitioning patterns, respectively, and 3 represents a collection of other partitioning patterns.

[0075] In S32, the specific process of predicting the PU partition mode according to the prediction class label of the CU partition depth is as follows: starting from the top depth of the quadtree encoding, the PU partition mode is predicted for the CU that does not need further partitioning from top to bottom at each depth level.

[0076] Specifically, the prediction workflow of the CU partition depth is as follows: starting from the top depth (depth=0) of the quadtree coding, such as a CU with a size of m×m pixels, for CUs that do not need to be partitioned, PU mode prediction is performed; if the CU needs to be further split, a new round of CU and PU prediction is performed at the new depth level (depth=depth+1, and m=m / 2) until the CU depth depth=3, after which the remaining necessary RDO processes are performed. In short, compared with the recursive RDO process, top-down prediction helps to realize the selection of CU partition depth and PU partition mode earlier, thereby reducing the complexity of quadtree coding. In this case, a partition depth prediction model with high prediction accuracy is the key to affecting the performance of the proposed fast DVC transcoding.

[0077] In one application of the present application, in order to demonstrate the beneficial effects of the transcoding technology of the present application, we compare it with the advanced DVC fast transcoding technology (transcoding technology based on SVM+feature selection and transcoding technology based on Naive Bayes), focusing on transcoding speed and compression performance. It should be noted that the transcoding technology based on Naive Bayes can only work at the CU level, that is, it can only predict the CU division depth, while the technology of the present application and the transcoding technology based on SVM+feature selection can run at both the CU and PU levels. In addition to predicting the division of the CU, it can also predict the PU division mode for CUs that are no longer divided further. In terms of experimental configuration, the transcoding algorithm is applied to HM16.5 using a low-latency P-based profile. The group of pictures consists of 8 frames and follows the IPP structure. In order to maintain the matching of video reconstruction quality, the quantization parameters (QP) of DVC and HEVC are {3, 4, 5, 6} and {18, 22, 26, 30}, respectively. In the proposed lightweight deep learning network, the number of categories is set to 2 for CU partition depth prediction, while the number of categories is set to 4 for PU partition mode prediction.

[0078] To clearly present the performance comparison results, Table 1 shows the comparison results of transcoding performance at the CU level. Cascade transcoding, which performs DVC decoding and HEVC encoding sequentially, is used as the performance comparison baseline. Delta bit rate (BD-BR) is used to measure rate-distortion (RD) performance. ΔT (%) defined in the following formula is used to measure transcoding speed:

[0079]

[0080] Among them, T anchor and T fastThey represent the HEVC encoding time in cascade transcoding and fast DVC to HEVC transcoding, respectively.

[0081] Table 1 Comparison of RD performance and transcoding speed of CU-level transcoding

[0082]

[0083] Table 1 shows that at the CU level, the average RD performance of the transcoding technology of the present application is better than that of the reference technology, although the cost is a slight decrease in transcoding speed due to the introduction of a lightweight network. Table 2 shows that at the PU level, the technology of the present application is better than the reference technology in both RD performance and transcoding speed. Specifically, the average BD-BR is reduced to 2.16%, while the average transcoding speed is increased by 60.97%. The enhanced transcoding performance is mainly attributed to the fact that the proposed lightweight deep learning model can deeply explore the correlation between temporal and spatial features through a reasonable network architecture and feature fusion, thereby more effectively exploring potential patterns and features from the data and achieving more accurate prediction of CU partition depth and PU partition mode. Accurate prediction helps to improve the compression performance of transcoding and also helps expand the prediction range of PU partition mode to the three most common modes. Therefore, compared with the reference technology, the present application further reduces the redundant steps of selecting the PU partition mode and speeds up the transcoding speed.

[0084] Table 2 Comparison of RD performance and transcoding speed of PU-level transcoding

[0085]

[0086] In order to further improve the DVC transcoding efficiency, this application adopts a more advanced algorithm to achieve efficient feature extraction and utilization, and proposes a low-power video transcoding method based on deep learning; specifically, through an innovative feature fusion mechanism, the spatial-temporal correlation of the initial features is fully explored at the global and local levels, and the prediction accuracy of the CU partition depth and PU mode partition is improved, thereby accelerating the DVC to HEVC transcoding speed while improving the compression performance of the DVC to HEVC transcoding.

[0087] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A low-power video transcoding method based on deep learning, characterized in that: include: S1. Data preparation: Transcode DVC-encoded video to HEVC-encoded video, and mark the initial feature vector according to the CU division depth and PU division mode during the transcoding process to obtain a dataset; S2. Model construction: Build a lightweight deep learning model and train it based on the dataset to obtain a partitioning depth prediction model; S3, partition prediction: the DVC video frame feature vector is processed and input into the partition depth prediction model to obtain the best prediction selection of CU partition depth and PU partition mode; S4. Video coding: HEVC encoding is performed on the DVC video frame according to the predicted best selection of CU partition depth and PU partition mode.

2. The low-power video transcoding method based on deep learning according to claim 1, characterized in that S1 includes: S11, decoding the DVC encoded video to obtain a reconstructed video frame and an initial feature vector; S12, performing HEVC transcoding on the reconstructed video frame to obtain real data of CU partition depth and PU partition mode; S13. Label the initial feature vector according to the real data of the CU and PU partitioning patterns to obtain a data set.

3. The low-power video transcoding method based on deep learning according to claim 1, characterized in that S3 include: S31, after processing the DVC video frame feature vector, input it into the partition depth prediction model to obtain the predicted class label of the CU partition depth; S32: Predict the PU partition mode according to the prediction class label of the CU partition depth.

4. The low-power video transcoding method based on deep learning according to claim 3, characterized in that: S31 includes: S311, extracting a luminance image containing intra-frame correlation and a residual image containing inter-frame correlation from a feature vector of a DVC video frame to be transcoded, and inputting the images into a partitioned depth prediction model; S312, respectively performing compression and excitation on the luminance image containing intra-frame correlation and the residual image containing inter-frame correlation to extract a low-dimensional feature map; S313: Perform convolution processing on the obtained low-dimensional feature map after feature fusion to obtain a higher-dimensional feature space and obtain the predicted class label of the CU partition depth.

5. The low-power video transcoding method based on deep learning according to claim 4, characterized in that: The feature fusion process performed in S313 is expressed as: in, represents the feature fusion result on the cth channel, c∈{1,2,...,C} represents the cth channel, C represents the number of output channels, u c and z c Respectively represent the values of the low-dimensional feature map corresponding to the brightness image before feature fusion and the low-dimensional feature map corresponding to the residual image on the cth channel, α c and β c are all weight matrices.

6. The low-power video transcoding method based on deep learning according to claim 5, characterized in that: α c and β c The value at position (i, j) is represented as: in, and They are control parameters and The value at position (i,j), and They are λ α and λ β Parameter matrix on the cth channel, controlling parameter λ α and λ β The expression is: l a =W2δ(W1U) l β =W2δ(W1Z) Among them, δ() represents the ReLU function, W1 and W2 are the weights of the 1×1 convolutional layer, and U and Z represent the output of the changes to the brightness image and residual image before feature fusion, respectively.

7. The low-power video transcoding method based on deep learning according to claim 3, characterized in that: The prediction class labels of the CU partition depth include needing further partitioning and not needing further partitioning. For the CU that does not need further partitioning, the PU partition mode prediction is performed.

8. The low-power video transcoding method based on deep learning according to claim 7, characterized in that: In S32, the specific process of predicting the PU partition mode according to the prediction class label of the CU partition depth is as follows: starting from the top depth of the quadtree encoding, the PU partition mode is predicted for the CU that does not need further partitioning from top to bottom at each depth level.

Images