Video coding CU block classification method based on lightweight deep learning model and medium

Abstract

The invention requests to protect a video coding CU block classification method based on a lightweight deep learning model, and is suitable for the field of video coding. The method comprises the following steps: firstly, carrying out block processing on an input image, dividing the input image into a plurality of 128 * 128-pixel image blocks, and extracting brightness information of each image block; after normalization processing, brightness data is used as input of a deep learning model and is used for generating classification probabilities of multiple categories. For the subunits of each image block, the probabilities of different categories are compared pixel by pixel, and a category label with the highest probability is allocated to the subunits. And then, based on the classification result of the subunits, counting the number of each category in each CU block, and allocating a corresponding type to the CU block according to statistical information. The CU block can belong to multiple categories simultaneously, and the final category of the CU block is determined through bitwise operation. According to the method, through accurate CU block classification, the coding efficiency is improved, data support is provided for subsequent mode division and selection, and the method is suitable for video coding application scenes with high real-time performance requirements.

Description

Technical Field

[0001] The present invention belongs to the field of video coding, and more specifically, is a coding unit (CodingUnit, CU) block classification method suitable for video coding, which can be applied to video coding scenarios with high real-time requirements. Background Art

[0002] With the development of Internet applications, users' requirements for video quality continue to increase, and high-definition and ultra-high-definition videos have begun to be widely used. The consumption of various types of videos, such as natural scene videos shot by cameras, screen content such as computer-generated text animations, and videos with moving text and graphics, has increased significantly. In application scenarios such as distance education and business meetings, the transmission demand for screen content data has increased significantly. The image characteristics of the screen content coding (Screen Content Coding, SCC) scene are different from those of natural scenes, and mainly include static or semi-static high-frequency content such as graphics, text, and UI elements. In order to meet this diverse video content demand, in 2016, the Joint Collaboration Team on Video Coding (JCT-VC) released the Screen Content Coding (Screen Content Coding, SCC) standard based on the high-efficiency video coding HEVC, namely HEVC-SCC. For screen content, technologies such as IntraBlock Copy (IBC), Palette (PLT), Adaptive Motion Vector Resolution (AMVP), and Adaptive Color Transform (ACT) have been added. In 2020, the Joint Video Experts Team (JVET) developed a new generation of video coding standard VVC, namely H.266 / VVC. VVC inherits the screen content coding tools in HEVC-SCC. Compared with HEVC, VVC further improves coding efficiency by supporting advanced coding technologies such as flexible CU block division.

[0003] In order to improve coding efficiency, VVC adopts technologies such as CU block division structure based on Quad-tree Plus Multi-type Tree (QTMT), Multiple Reference Line (MRL) prediction, Intra Sub-Partitions Mode (ISP), etc., which allows adaptive selection of CU block division and encoding methods based on image features. This division method shows high coding efficiency when dealing with natural scene videos, but in screen content encoding, due to the special characteristics of screen content, such as high-frequency edges, repeated patterns, and static areas, the encoder faces significant computational challenges. Especially in real-time applications, how to effectively distinguish between natural content CU blocks and screen content CU blocks while ensuring coding quality has become a key issue.

[0004] CU block classification methods based on deep learning have gradually become a research hotspot. Using deep learning models such as convolutional neural networks (CNN), CU blocks can be divided into natural content CUs (CC class) and screen content CUs (including A class, TGM class, and Mixed class), thereby reducing encoding complexity. Specifically, the deep learning model can identify the texture features of the CU block, predict through classification whether the CU block belongs to the natural scene or screen content, and adjust the encoding strategy accordingly to improve encoding efficiency and compression effect. Therefore, applying deep learning to the field of video coding is an important research direction at present.

[0005] The present invention proposes a video coding CU block classification method based on a lightweight deep learning model. Through the prediction ability of the deep learning model, the CU block is automatically classified into a natural content CU block or a screen content CU block, and the corresponding encoding method is adopted, which not only meets the encoding requirements of the screen content, but also effectively reduces the computational complexity of the encoder when processing the screen content, providing technical support for the real-time application of high-quality video encoding.

[0006] After searching, the application publication number CN114286093A is a fast video encoding method based on deep neural network. The present invention includes a CU partitioning module based on deep neural network and a PU mode selection module based on neighborhood correlation; when the CU block is intra-coded, the PU mode selection module based on neighborhood correlation is first used for optimization, and the number of candidate modes calculated by RDO is reduced by the prediction results of the lightweight HCT model; after the PU mode selection is completed, the encoder will make a CU block depth judgment to determine whether the CU block is divided. At this time, the CU partitioning module based on deep neural network is optimized, and the prediction results are obtained from the HCT model to determine whether to terminate the division in advance. Otherwise, continue to divide the sub-CU blocks downward, and continue to perform PU mode selection and CU block partitioning judgment. The present invention reduces the complexity of CU recursive partitioning, simplifies the selection process of intra-frame prediction mode, and effectively improves the time efficiency of HEVC encoding.

[0007] Patent CN114286093A proposes a fast video coding method based on deep neural network, which reduces the number of RDO calculation candidate modes through PU neighborhood correlation and lightweight HCT model, and uses HCT model prediction to terminate CU block division in advance, thereby improving the time efficiency of HEVC coding. However, this method is only applicable to HEVC coding, and has poor compatibility with new generation coding standards such as VVC, and its universality and application scope are limited. If CN114286093A is combined with the method of the present invention, the video coding process can be further accelerated and the application scenarios can be expanded. For example, the lightweight CNN model of the present invention is first used to classify the CU block type, and then the fast coding method of CN114286093A is used according to different CU types (natural content CU or screen content CU), which can more effectively speed up the coding speed, and is particularly suitable for mixed video coding that contains both natural and screen content, such as conferences and distance education. Summary of the invention

[0008] The present invention aims to solve the above problems of the prior art. A video coding CU block classification method based on a lightweight deep learning model is proposed. The technical solution of the present invention is as follows:

[0009] A video coding CU block classification method based on a lightweight deep learning model comprises the following steps:

[0010] S1. Design a lightweight convolutional neural network (CNN) model for the classification of coding unit (CU) types.

[0011] S2. Collect video sequence datasets for training CNN models. The video sequences cover A (animation), CC (camera content) and TGM (text and graphics with motion) categories. Divide the collected datasets into 20% of the samples as test sets and 80% of the samples as training sets.

[0012] S3, preprocessing the collected video sequence set, annotating different types of video data, and constructing a training data set for training the CNN model;

[0013] S4, based on the training data set constructed in step S3, use machine learning methods to train the designed CNN model, use back propagation algorithm to update model parameters, use validation set to evaluate the performance of the trained model, and optimize the accuracy of the model;

[0014] S5. Deploy the CNN model trained in step S4 to the high-efficiency video coding HEVC and general video coding VVC video encoders to classify the CU type of the video to be encoded.

[0015] Furthermore, in the step S1, a lightweight convolutional neural network (CNN) model is designed for the classification of coding unit CU types, specifically including: the lightweight CNN model adopts a symmetrical encoder-decoder structure, and its encoder gradually extracts features through 6 convolutional layers, batch normalization layers, and ReLU activation functions, and reduces the input 128×128 single-channel image to 2×2×512; the bottleneck layer further compresses the feature map to 1×1×1024 to extract deep features; the decoder gradually restores the size of the feature map through 5 deconvolution layers, and outputs a classification result of 32×32×3; the model does not use jump connections to avoid additional channel growth; finally, the output layer uses 1×1 convolution and Softmax function to classify each pixel to complete the classification of CU types.

[0016] Furthermore, in step S2, a video sequence data set for training the CNN model is collected, and the video sequences cover categories A, CC and TGM. Category A video sequences are computer-generated animation videos; category CC video sequences are videos shot by a camera; and category TGM video sequences are videos with moving text and graphics. The collected video sequences are mainly derived from the Internet and the test sequence set used by the video coding standard.

[0017] Furthermore, in step S3, the collected video sequence set is preprocessed. Different types of video data are labeled, and the collected video sequences are grayscaled, normalized, and cropped through the Python platform. The following steps are specifically included: first, the selected video sequence is grayscaled, and the color image is converted into a single-channel grayscale image to simplify the data and improve processing efficiency; second, each frame of the image is cropped into an image block of 128×128 pixels to ensure the standardization of the input data; then, the cropped image blocks are content-annotated and divided into CC, A, and TGM categories. The preprocessing of this step provides standardized data input for the training of the CNN model.

[0018] Furthermore, in step S4, the designed CNN model is trained using a machine learning method, and the model parameters are updated using a back propagation algorithm. The loss function used for training the designed CNN model is:

[0019]

[0020] Among them, Loss is the loss; C is the total number of categories; y i is the true label; p i is the probability of belonging to category i predicted by the model;

[0021] In the back propagation algorithm, for each layer parameter θ l The gradient calculation formula is:

[0022]

[0023] Among them, a l+1 is the activation output of layer l+1; z l represents the linear transformation output of the lth layer; θ l Represents the parameters of the lth layer.

[0024] The parameter update of each layer uses the Adam optimizer to update the parameters, and the update formula is:

[0025]

[0026] Where η is the learning rate.

[0027] Furthermore, in step S5, the trained CNN model is deployed to the HEVC and VVC video encoders to classify the CU block types of the video to be encoded. The specific process is as follows:

[0028] S51. Calculate the number of 128×128 image blocks of the current frame to be encoded according to the resolution of the frame. Read an image block from left to right and from top to bottom, and proceed to step S52. If the read image block is less than 128×128, proceed to step S59.

[0029] S52, extracting and saving the brightness information of each pixel of the 128×128 image block; normalizing the brightness value of each pixel to a value in the range of 0 to 1 to meet the requirements of input standardization of the neural network model, and proceeding to step S53;

[0030] S53, load the pre-trained CNN model, convert the normalized brightness information into a tensor in the model input format, and then input it into the CNN model for reasoning, and enter step S54;

[0031] S54, after model inference, the classification probability of each image block is generated, that is, the probability values ​​of three channels, corresponding to three different categories respectively; the array is stored using a row-first Z-shaped indexing method, and the index of each image block is obtained by multiplying the row number of the image block by the width of the image block plus the column number; the output result is stored in the designated memory for subsequent access and classification operations, and then enters step S55;

[0032] S55, check whether the current CU block to be encoded is a luminance component and the CU position is non-boundary, if yes, go to step S56; if no, go to step S59;

[0033] S56, traverse the 4×4 pixel sub-regions of each CU block according to the classification probability output by the model; read the classification probability of each sub-region in turn, compare the probability values ​​of the three categories, select the category label with the highest probability, and assign the label to the 4×4 region; in this way, generate a preliminary category distribution map of the current CU block, and store the category label of each 4×4 region in a two-dimensional array for further statistical processing, and enter step S57;

[0034] S57. Calculate the final category label of the CU block using bit operations: first initialize the category count array and count the number of categories in the current CU block. Traverse all 4×4 sub-regions in the CU block, count the category label of each region into the corresponding category count, and accumulate them category by category; initialize the final category label variable of the CU block. Use bit operations to combine multiple category information. If a category appears in the CU block, add the flag bit of the category to the category label of the CU block through bit operations, so that each bit represents a category. In this way, when multiple categories appear, the category label will record all the category information that appears; after this step is completed, the category label of the CU block contains all the categories that appear in the current CU block, and enter step S58;

[0035] S58. Parse the content type according to the category label of the CU block. The category label of the CU block has expressed all the category information in the CU block through bit operations, and each bit corresponds to a category; by reading the status of each bit of the category label, the content type contained in the current CU block is determined, including category A, category CC, category TGM and category Mixed; the Mixed category here is a mixed type containing two or three of the categories A, category CC and category TGM; the parsed category label provides content category support for the subsequent encoding process, allowing the encoder to perform adaptive encoding according to the characteristics of the CU block. Go to step S59;

[0036] S59, end the current CU block type classification.

[0037] Furthermore, in step S57, the model outputs the category probability information of each CU block; after the output of the model is processed, the probability information is used to calculate the occurrence frequency of each category in each 4×4 pixel unit; the CU block type includes four categories: category A, category CC, category TGM and category Mixed; according to the occurrence of the category, different categories are encoded using bitwise operations, and this information is recorded in the CU category code to ensure that the category combination of the current CU block is efficiently stored and called.

[0038] Furthermore, in step S57, the calculation of the CU block category is implemented by a bitwise operation formula to determine the combination of each category in the CU block; the CU block category includes four categories: CC class, A class, TGM class and Mixed class, and the following formula is used to calculate the CU block category code:

[0039]

[0040] Among them, CU Category Encoding Indicates CU category; i indicates the index value of the category, 0 indicates CC category, 1 indicates A category, 2 indicates TGM category, and others indicate Mixed category; P c Represents a Boolean value, which is 1 if the category exists, otherwise it is 0; 1<<i means shifting 1 left to the position corresponding to the category.

[0041] A storage medium stores a computer program internally, wherein when the computer program is read by a processor, any of the above-mentioned video coding CU block classification methods based on a lightweight deep learning model is executed.

[0042] The advantages and beneficial effects of the present invention are as follows:

[0043] The present invention designs a lightweight convolutional neural network (CNN) model to classify the current CU block in the coded frame to distinguish different types of video content (natural content and screen content). First, the CNN model extracts and analyzes the features of the CU block to determine its type; for screen content (class A, TGM, Mixed) and natural content (class CC), the model can effectively distinguish the texture and structural characteristics of different types of CU blocks. The classification results provide a basis for the subsequent encoder to select the most suitable CU block partitioning mode and encoding tools, thereby realizing an adaptive encoding strategy.

[0044] In order to evaluate the encoding acceleration performance of the method proposed in the present invention, the present invention performs performance tests on the official encoder VTM-17.0. The test standard follows the official Common Test Condition (CTC) specified for the Standard Dynamic Range (SDR) video sequence. One sequence is selected for each type of video for testing, and the All Intra (AI) encoding configuration is used to perform encoding performance tests under four quantization parameter (QP) values ​​{37, 32, 27, 22}. Taking BD-BR ( delta bit-rate) and ΔT are used as performance evaluation indicators of the acceleration algorithm. BD-BR measures the loss of coding efficiency. The larger its value, the greater the loss of coding efficiency. ΔT represents the time saved by the algorithm. Its calculation formula is:

[0045]

[0046] Among them, T anchor represents the encoding time of the original encoder, T proposed It represents the encoding time of the encoder using the model proposed in the present invention.

[0047] Table 1 shows the encoding performance of each type of video sequence. Proposed in Table 1 represents the method of the present invention, and Ref is the algorithm in the reference (Wang D, Yu J, Lu X, et al. Fast Mode and CU Splitting Decision for IntraPrediction in VVC SCC [J]. IEEE Transactions on Broadcasting, 2024.). It can be seen that the method of the present invention can reduce the encoding time by an average of 54.66%, while BD-BR only increases it by 1.30%. From the comparison results, it can be seen that the performance of the model in this paper is significantly better than the algorithm in the reference.

[0048] Table 1 Performance comparison between the present invention and the reference method

[0049]

[0050] Figure 1 The accuracy and loss curves during model training are shown. During model training, both the accuracy and loss curves show good convergence and stability. The accuracy curve shows that the accuracy of the training set and the test set gradually increases with the increase of training cycles, and increases rapidly within the first 100 epochs, and finally reaches a stable level of more than 90%, indicating that the model has good classification performance and no obvious overfitting. The loss curve drops rapidly at the beginning of training, verifying that the model effectively minimizes the error, and stabilizes after about 600 epochs, indicating that the model is close to the optimal state. The loss curves of the training and test sets are close to each other, further proving that the model has good generalization ability and can efficiently complete the CU block type classification task.

[0051] The innovation of the present invention mainly lies in claim 4 and claim 6. The method of claim 4 is adopted to provide high-quality training data for the CNN model through data preprocessing and label annotation. The method of claim 6 is adopted to classify the CU block type using the trained lightweight CNN model. The present invention selects different encoding modes for subsequent different types of CUs while ensuring that the video quality remains basically unchanged and the encoding output bit rate does not increase much, thereby effectively reducing the computational burden of the VVC encoder in complex content processing. By optimizing the CU type recognition process, the present invention is suitable for high-resolution video scenes such as high-definition, ultra-high-definition video and screen content encoding, and can improve encoding efficiency and shorten encoding time while meeting real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The present invention provides the accuracy and loss curves during the model training process of the preferred embodiment.

[0053] Figure 2 The present invention provides a preferred embodiment of a video coding CU block classification method based on a lightweight deep learning model.

[0054] Figure 3 It is a lightweight CNN model structure. DETAILED DESCRIPTION

[0055] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0056] The technical solution of the present invention to solve the above technical problems is:

[0057] Figure 2 The present invention provides a video coding CU block classification method based on a lightweight deep learning model. The method of the present invention comprises the following steps:

[0058] S1. Design a lightweight convolutional neural network (CNN) model for the classification of coding unit (CU) types.

[0059] S2. Collect video sequence datasets for training CNN models. The video sequences cover A (Animation), CC (Camera-captured Content) and TGM (Text and Graphics with Motion). Divide the collected datasets into 20% of the samples as test sets and 80% of the samples as training sets.

[0060] S3. Preprocess the collected video sequence set. Label different types of video data and construct a training data set for training the CNN model.

[0061] S4. Based on the training data set constructed in step S3, the designed CNN model is trained using a machine learning method, the model parameters are updated using a back-propagation algorithm, the performance of the trained model is evaluated using a validation set, and the accuracy of the model is optimized.

[0062] S5. Deploy the CNN model trained in step S4 to video encoders such as High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC), and classify the CU type of the video to be encoded for subsequent processing such as CU block division to improve coding efficiency.

[0063] Preferably, in step S1, a lightweight convolutional neural network (CNN) model is designed for the classification of coding unit (CU) types, specifically including: the lightweight CNN model adopts a symmetrical encoder-decoder structure. The encoder gradually extracts features through 6 convolutional layers, batch normalization layers, and ReLU activation functions, and reduces the input 128×128 single-channel image to 2×2×512; the bottleneck layer further compresses the feature map to 1×1×1024 to extract deep features; the decoder gradually restores the size of the feature map through 5 deconvolution layers, and outputs a 32×32×3 classification result. The model does not use jump connections to avoid additional channel growth, reduce the number of parameters, and make the model layers concise and efficient; finally, the output layer uses 1×1 convolution and Softmax function to classify each pixel and complete the classification of CU types.

[0064] Preferably, in step S2, a video sequence data set for training the CNN model is collected, and the video sequences include class A, class CC and class TGM. Class A video sequences are computer-generated animation videos; class CC video sequences are videos shot by a camera; and class TGM video sequences are videos with moving text and graphics. The collected video sequences are mainly derived from the Internet and the test sequence set used by the video coding standard.

[0065] Preferably, in step S3, the collected video sequence set is preprocessed. Different types of video data are labeled. The collected video sequence is preprocessed by grayscale, normalization and cropping through platforms such as Python, specifically including the following steps: first, the selected video sequence is grayscaled and the color image is converted into a single-channel grayscale image to simplify the data and improve processing efficiency; secondly, each frame of the image is cropped into an image block of 128×128 pixels to ensure the standardization of the input data; then, the cropped image blocks are content labeled and divided into CC, A and TGM categories. The preprocessing in this step provides standardized data input for the training of the CNN model.

[0066] Preferably, in step S4, the designed CNN model is trained using a machine learning method, and the model parameters are updated using a back propagation algorithm. The loss function used for training the designed CNN model is:

[0067]

[0068] Among them, Loss is the loss; C is the total number of categories; y i is the true label; p i is the probability of belonging to category i predicted by the model.

[0069] In the back propagation algorithm, for each layer parameter θ l The gradient calculation formula is:

[0070]

[0071] Among them, a l+1 is the activation output of layer l+1; z l represents the linear transformation output of the lth layer; θ l Represents the parameters of the lth layer.

[0072] The parameter update of each layer uses the Adam optimizer to update the parameters, and the update formula is:

[0073]

[0074] Where η is the learning rate.

[0075] Preferably, in step S5, the trained CNN model is deployed to a video encoder such as HEVC or VVC to classify the CU block type of the video to be encoded. The specific process is as follows:

[0076] S51. Calculate the number of 128×128 image blocks of the current frame to be encoded according to the resolution of the frame. Read an image block from left to right and from top to bottom, and proceed to step S52. If the read image block is less than 128×128, proceed to step S59.

[0077] S52, extract and save the brightness information of each pixel of the 128×128 image block. Normalize the brightness value of each pixel to a value in the range of 0 to 1 to meet the requirements of input standardization of the neural network model, and then proceed to step S53.

[0078] S53, load the pre-trained CNN model. Convert the normalized brightness information into a tensor in the model input format, and then input it into the CNN model for inference, and enter step S54.

[0079] S54, after model inference, the classification probability of each image block is generated, that is, the probability values ​​of three channels, corresponding to three different categories. The array is stored using a row-first Z-shaped indexing method, and the index of each image block is obtained by multiplying the row number of the image block by the width of the image block plus the column number. The output result is stored in the designated memory for subsequent access and classification operations, and then enters step S55.

[0080] S55, check whether the current CU block to be encoded is a luminance component and the CU position is non-boundary, if yes, go to step S56; if not, go to step S59.

[0081] S56. Traverse the 4×4 pixel sub-regions of each CU block according to the classification probability output by the model. Read the classification probability of each sub-region in turn, compare the probability values ​​of the three categories, select the category label with the highest probability, and assign the label to the 4×4 region. In this way, generate a preliminary category distribution map of the current CU block, and store the category label of each 4×4 region in a two-dimensional array for further statistical processing, and enter step S57.

[0082] S57. Use bit operations to calculate the final category label of the CU block: first initialize the category count array and count the number of categories in the current CU block. Traverse all 4×4 sub-areas in the CU block, count the category label of each area into the corresponding category count, and accumulate them category by category; initialize the final category label variable of the CU block. Use bit operations to combine multiple category information. If a category appears in the CU block, the flag bit of the category is added to the category label of the CU block through bit operations, so that each bit represents a category. In this way, when multiple categories appear, the category label will record all the category information that appears. After this step is completed, the category label of the CU block contains all the categories that appear in the current CU block, and enters step S58.

[0083] S58. Parse the content type according to the category label of the CU block. The category label of the CU block has expressed all the category information in the CU block through bit operations, and each bit corresponds to a category. By reading the status of each bit of the category label, the content type contained in the current CU block is determined, including Class A, Class CC, Class TGM and Class Mixed. The Mixed class here is a mixed type that includes two or three of the Class A, Class CC and Class TGM. The parsed category label provides content category support for the subsequent encoding process, enabling the encoder to perform adaptive encoding based on the characteristics of the CU block. Go to step S59.

[0084] S59, end the current CU block type classification.

[0085] Preferably, in step S57, the model outputs the category probability information of each CU block. After the output of the model is processed, the probability information is used to calculate the frequency of occurrence of each category in each 4×4 pixel unit. The CU block type includes four categories: A, CC, TGM and Mixed. According to the occurrence of the category, different categories are encoded using bitwise operations, and this information is recorded in the CU category code to ensure that the category combination of the current CU block is efficiently stored and called.

[0086] In step S57, the calculation of the CU block category is implemented by a bitwise operation formula to determine the combination of each category in the CU block. The CU block category includes four categories: CC, A, TGM and Mixed. The following formula is used to calculate the CU block category code:

[0087]

[0088] Among them, CU Category Encoding Indicates CU category; i indicates the index value of the category (0 indicates CC category, 1 indicates A category, 2 indicates TGM category, and others indicate Mixed category); P c Represents a Boolean value, which is 1 if the category exists, otherwise it is 0; 1<<i means shifting 1 left to the position corresponding to the category.

[0089] A storage medium stores a computer program internally, wherein when the computer program is read by a processor, any of the above-mentioned video coding CU block classification methods based on a lightweight deep learning model is executed.

[0090] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions.

[0091] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0092] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0093] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A video coding CU block classification method based on a lightweight deep learning model, characterized in that: The following steps are involved: S1. Design a lightweight convolutional neural network (CNN) model for the classification of coding unit (CU) types. S2. Collect video sequence datasets for training CNN models. The video sequences cover A animation, CC camera shooting content, and TGM moving text and graphics. The collected datasets are divided into 20% of the samples as test sets and 80% of the samples as training sets. S3, preprocessing the collected video sequence set, annotating different types of video data, and constructing a training data set for training the CNN model; S4, based on the training data set constructed in step S3, use machine learning methods to train the designed CNN model, use back propagation algorithm to update model parameters, use validation set to evaluate the performance of the trained model, and optimize the accuracy of the model; S5. Deploy the CNN model trained in step S4 to the high-efficiency video coding HEVC and general video coding VVC video encoders to classify the CU type of the video to be encoded.

2. The video coding CU block classification method based on a lightweight deep learning model according to claim 1, characterized in that: In the step S1, a lightweight convolutional neural network (CNN) model is designed for the classification of coding unit (CU) types, specifically comprising: the lightweight CNN model adopts a symmetrical encoder-decoder structure, wherein the encoder gradually extracts features through 6 convolutional layers, batch normalization layers, and ReLU activation functions, and reduces the input 128×128 single-channel image to 2×2×512; the bottleneck layer further compresses the feature map to 1×1×1024 to extract deep features; the decoder gradually restores the size of the feature map through 5 deconvolution layers, and outputs a classification result of 32×32×3; the model does not use jump connections to avoid additional channel growth; finally, the output layer uses 1×1 convolution and Softmax function to classify each pixel and complete the classification of the CU type.

3. The video coding CU block classification method based on a lightweight deep learning model according to claim 1, characterized in that: In the step S2, a video sequence data set for training the CNN model is collected, and the video sequences cover categories A, CC and TGM. Category A video sequences are computer-generated animation videos; category CC video sequences are videos shot by a camera; and category TGM video sequences are videos with moving text and graphics. The collected video sequences are mainly derived from the Internet and the test sequence set used by the video coding standard.

4. The video coding CU block classification method based on a lightweight deep learning model according to claim 1, characterized in that: In step S3, the collected video sequence set is preprocessed. Different types of video data are labeled, and the collected video sequences are grayscaled, normalized, and cropped through the Python platform. The following steps are specifically included: first, the selected video sequence is grayscaled and the color image is converted into a single-channel grayscale image to simplify the data and improve the processing efficiency; second, each frame of the image is cropped into an image block of 128×128 pixels to ensure the standardization of the input data; then, the cropped image blocks are content-annotated and divided into CC, A, and TGM categories. The preprocessing of this step provides standardized data input for the training of the CNN model.

5. The video coding CU block classification method based on a lightweight deep learning model according to claim 1, characterized in that: In step S4, the designed CNN model is trained using a machine learning method, and the model parameters are updated using a back propagation algorithm. The loss function used for training the designed CNN model is: Among them, Loss is the loss; C is the total number of categories; y i is the true label; p i is the probability of belonging to category i predicted by the model; In the back propagation algorithm, for each layer parameter θ l The gradient calculation formula is: Among them, a l+1 is the activation output of layer l+1; z l represents the linear transformation output of the lth layer, θ l Represents the parameters of the lth layer; The parameter update of each layer uses the Adam optimizer to update the parameters, and the update formula is: Where η is the learning rate.

6. The video coding CU block classification method based on a lightweight deep learning model according to claim 1, characterized in that: In step S5, the trained CNN model is deployed to the HEVC and VVC video encoders to classify the CU block types of the video to be encoded. The specific process is as follows: S51. Calculate the number of 128×128 image blocks of the current frame to be encoded according to the resolution of the frame. Read an image block from left to right and from top to bottom, and proceed to step S52. If the read image block is less than 128×128, proceed to step S59. S52, extracting and saving the brightness information of each pixel of the 128×128 image block; normalizing the brightness value of each pixel to a value in the range of 0 to 1 to meet the requirements of input standardization of the neural network model, and proceeding to step S53; S53, load the pre-trained CNN model, convert the normalized brightness information into a tensor in the model input format, and then input it into the CNN model for reasoning, and enter step S54; S54, after model inference, the classification probability of each image block is generated, that is, the probability values ​​of three channels, which correspond to three different categories respectively; the array is stored using a row-first Z-shaped indexing method, and the index of each image block is obtained by multiplying the row number of the image block by the width of the image block plus the column number; The output result is stored in the designated memory for subsequent access and classification operations, and the process proceeds to step S55; S55, check whether the current CU block to be encoded is a luminance component and the CU position is non-boundary, if yes, go to step S56; if no, go to step S59; S56, traverse the 4×4 pixel sub-regions of each CU block according to the classification probability output by the model; read the classification probability of each sub-region in turn, compare the probability values ​​of the three categories, select the category label with the highest probability, and assign the label to the 4×4 region; in this way, generate a preliminary category distribution map of the current CU block, and store the category label of each 4×4 region in a two-dimensional array for further statistical processing, and enter step S57; S57. Calculate the final category label of the CU block using bit operations: first initialize the category count array and count the number of categories in the current CU block. Traverse all 4×4 sub-regions in the CU block, count the category label of each region into the corresponding category count, and accumulate them category by category; initialize the final category label variable of the CU block. Use bit operations to combine multiple category information. If a category appears in the CU block, add the flag bit of the category to the category label of the CU block through bit operations, so that each bit represents a category. In this way, when multiple categories appear, the category label will record all the category information that appears; after this step is completed, the category label of the CU block contains all the categories that appear in the current CU block, and enter step S58; S58. Parse the content type according to the category label of the CU block. The category label of the CU block has expressed all the category information in the CU block through bit operations, and each bit corresponds to a category; by reading the status of each bit of the category label, the content type contained in the current CU block is determined, including category A, category CC, category TGM and category Mixed; the Mixed category here is a mixed type containing two or three of the categories A, category CC and category TGM; the parsed category label provides content category support for the subsequent encoding process, enabling the encoder to perform adaptive encoding according to the characteristics of the CU block. Go to step S59; S59. End the current CU block type classification.

7. The video coding CU block classification method based on a lightweight deep learning model according to claim 6 is characterized in that: In step S57, the model outputs the category probability information of each CU block; after the output of the model is processed, the probability information is used to calculate the occurrence frequency of each category in each 4×4 pixel unit; the CU block type includes four categories: category A, category CC, category TGM and category Mixed; according to the occurrence of the category, different categories are encoded using bitwise operations, and this information is recorded in the CU category code to ensure that the category combination of the current CU block is efficiently stored and called.

8. The video coding CU block classification method based on a lightweight deep learning model according to claim 7, characterized in that: In step S57, the calculation of the CU block category is implemented by a bitwise operation formula to determine the combination of each category in the CU block; the CU block category includes four categories: CC class, A class, TGM class and Mixed class. The following formula is used to calculate the CU block category code: Among them, CU CategoryEncoding Indicates CU category; i indicates the index value of the category, 0 indicates CC category, 1 indicates A category, 2 indicates TGM category, and others indicate Mixed category; P c Represents a Boolean value, which is 1 if the category exists, otherwise it is 0; 1<<i means shifting 1 left to the position corresponding to the category.

9. A storage medium storing a computer program, characterized in that: When the computer program is read by the processor, the video coding CU block classification method based on the lightweight deep learning model described in any one of claims 1 to 8 is executed.

Images