Video coding cu block classification method based on lightweight deep learning model and medium

Abstract

The application claims a video coding CU block classification method based on a lightweight deep learning model, which is suitable for the field of video coding. The method first performs block processing on the input image, divides it into multiple 128x128 pixel image blocks, and extracts the brightness information of each image block. After normalization processing, the brightness data is used as the input of the deep learning model to generate classification probabilities of multiple categories. For each subunit of the image block, the probabilities of different categories are compared pixel by pixel, and the category label with the highest probability is assigned to it. Subsequently, based on the classification results of the subunits, the number of each category in each CU block is counted, and the corresponding type of the CU block is assigned according to the statistical information. The CU block can belong to multiple categories at the same time, and the final category is determined by bitwise operation. Through accurate CU block classification, the application not only improves the coding efficiency, but also provides data support for subsequent mode division and selection, and is suitable for video coding application scenarios with high real-time requirements.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of video coding, and more particularly to a coding unit (CU) block classification method applicable to video coding, which can be applied to video coding scenarios with high real-time requirements. BACKGROUND

[0002] With the development of Internet applications, users' requirements for video quality are constantly improving, and high-definition and ultra-high-definition videos have begun to be widely used. Natural scene videos captured by cameras, screen content generated by computers, such as text animations, and various videos with moving text and graphics have seen a large-scale increase in consumption. In application scenarios such as remote education and business meetings, the transmission demand for screen content data volume has significantly increased. The image features of screen content coding (SCC) scenarios are different from those of natural scenes, mainly including graphics, text, UI elements, and other static or semi-static high-frequency content. In order to meet the diverse needs of video content, in 2016, the Joint Collaboration Team on Video Coding (JCT-VC) released the screen content coding standard (SCC) based on the High Efficiency Video Coding (HEVC) extension, i.e., HEVC-SCC. Intra block copy (IBC), palette mode (PLT), adaptive motion vector resolution (AMVP), and adaptive color transform (ACT) are added for screen content. In 2020, the Joint Video Experts Team (JVET) developed a new generation of video coding standard VVC, i.e., H.266 / VVC. VVC inherits the screen content coding tools in HEVC-SCC. Compared with HEVC, VVC further improves coding efficiency by supporting flexible CU block division methods and other advanced coding technologies.

[0003] To improve coding efficiency, VVC adopts the CU block partition structure based on Quad-tree Plus Multi-type Tree (QTMT), Multiple Reference Line (MRL) prediction, Intra Sub-Partitions Mode (ISP) and other technologies, allowing adaptive selection of CU block partition and coding mode according to image characteristics. This partition method shows high coding efficiency when dealing with natural scene videos, but in screen content coding, due to the special features of screen content such as high-frequency edges, repetitive patterns and static regions, the encoder faces significant computational challenges. Especially in real-time applications, how to effectively distinguish natural content CU blocks and screen content CU blocks while ensuring coding quality has become a key issue.

[0004] CU block classification methods based on deep learning have gradually become a research hotspot. Using deep learning models such as convolutional neural networks (CNN), CU blocks can be classified into natural content CUs (CC class) and screen content CUs (including A class, TGM class and Mixed class), thereby reducing coding complexity. Specifically, deep learning models can identify the texture features of CU blocks, predict whether the CU block belongs to a natural scene or screen content through classification, and adjust the coding strategy accordingly to improve coding efficiency and compression effect. Therefore, applying deep learning to the field of video coding is an important research direction at present.

[0005] The present application proposes a video coding CU block classification method based on a lightweight deep learning model. Through the prediction ability of the deep learning model, the CU blocks are automatically classified into natural content CU blocks or screen content CU blocks, and the corresponding coding method is adopted, which not only meets the coding needs of screen content, but also effectively reduces the computational complexity of the encoder when processing screen content, providing technical support for real-time applications of high-quality video coding.

[0006] After searching, application publication number CN114286093A, a fast video coding method based on deep neural network. The invention includes a CU division module based on deep neural network, a PU mode selection module based on neighborhood correlation; CU block will first pass through PU mode selection calculation rate distortion cost when intra coding, at this time, first use the PU mode selection module based on neighborhood correlation to optimize, reduce the number of candidate modes of RDO calculation through the prediction result of the lightweight HCT model; after PU mode selection, the encoder will make CU block depth decision, judge whether the CU block is divided, at this time, the CU division module based on deep neural network is optimized, the prediction result is obtained from the HCT model to judge whether to terminate the division in advance. Otherwise, continue to divide the sub-CU block, and continue to select PU mode and CU block division. The invention reduces the complexity of CU recursive division, simplifies the selection process of intra prediction mode, and effectively improves the time efficiency of HEVC coding.

[0007] Patent CN114286093A proposes a fast video coding method based on deep neural network, which reduces the number of RDO calculation candidate modes through PU neighborhood correlation and lightweight HCT model, and uses HCT model prediction to terminate CU block division in advance, thereby improving the time efficiency of HEVC coding. However, this method is only applicable to HEVC coding, and has poor compatibility with new generation coding standards such as VVC, and the universality and application range are limited. If CN114286093A is combined with the method of the present invention, the video coding process can be further accelerated and the application scenarios can be expanded. For example, first use the lightweight CNN model of the present invention to classify the CU block type, then use the fast coding method of CN114286093A according to different CU types (natural content CU or screen content CU), which can effectively speed up the coding speed, especially suitable for mixed video coding containing natural and screen content such as conference and remote education. SUMMARY

[0008] The present invention aims to solve the problems of the above prior art. A video coding CU block classification method based on lightweight deep learning model is proposed. The technical solution of the present invention is as follows:

[0009] A video coding CU block classification method based on lightweight deep learning model, comprising the following steps:

[0010] S1, design a lightweight convolutional neural network (CNN) model for coding unit (CU) type classification;

[0011] S2, collect a video sequence dataset for training the CNN model, the video sequence covers A (animation) class, CC (camera shooting content) class and TGM (text and graphics with motion) class, divide the collected dataset, 20% of the samples as a test set and 80% of the samples as a training set;

[0012] S3, pre-process the collected video sequence set, label different types of video data and construct a training dataset for training the CNN model;

[0013] S4, based on the training dataset constructed in step S3, train the designed CNN model using a machine learning method, update the model parameters using a back propagation algorithm, evaluate the performance of the trained model using a validation set and optimize the accuracy of the model;

[0014] S5, deploy the trained CNN model in step S4 to a high-efficiency video coding (HEVC) and general video coding (VVC) video encoder and classify the CU types of the video to be encoded.

[0015] Further, in step S1, a lightweight convolutional neural network (CNN) model is designed for CU type classification, which specifically includes: the lightweight CNN model adopts a symmetric encoder-decoder structure, the encoder extracts features step by step through 6 convolutional layers, batch normalization layers and ReLU activation functions, and reduces the input 128x128 single-channel image to 2x2x512; the bottleneck layer further compresses the feature map to 1x1x1024 to extract deep-level features; the decoder gradually restores the size of the feature map through 5 deconvolutional layers and outputs a 32x32x3 classification result; the model does not use skip connection to avoid additional channel growth; finally, the output layer uses 1x1 convolution and Softmax function to classify each pixel and complete the classification of the CU type.

[0016] Further, in step S2, a video sequence dataset for training the CNN model is collected, the video sequence covers A class, CC class and TGM class, the A class video sequence is a computer-generated animation video, the CC class video sequence is a camera-shot video, and the TGM class video sequence is a text and graphics with motion video, and the collected video sequence is mainly from the Internet and the test sequence set used by the video coding standard.

[0017] Further, in step S3, the collected video sequence set is preprocessed. Different types of video data are labeled, and the collected video sequence is preprocessed by Python platform for greying, normalization and cropping. The specific steps include: first, the selected video sequence is processed for greying, and the color image is converted into a single-channel gray image to simplify the data and improve the processing efficiency; second, each frame of image is cropped into an image block with a size of 128x128 pixels to ensure the standardization of input data; then, the cropped image block is labeled for content, which is divided into CC class, A class and TGM class. The preprocessing of this step provides standardized data input for the training of the CNN model.

[0018] Further, in step S4, the designed CNN model is trained using a machine learning method, and the model parameters are updated using a back propagation algorithm. The loss function used for training the designed CNN model is:

[0019]

[0020] wherein Loss is the loss; C is the total number of classes; y i is the real label; p i is the probability of the model prediction belonging to the class i.

[0021] In the back propagation algorithm, the gradient calculation formula of each layer parameter θ l is:

[0022]

[0023] wherein a l+1 is the activation output of l+1 layer; z l represents the linear transformation output of the lth layer; θ l represents the parameter of the lth layer.

[0024] The parameter update of each layer uses the Adam optimizer to update the parameters, and the update formula is:

[0025]

[0026] wherein η is the learning rate.

[0027] Further, in step S5, the trained CNN model is deployed into the HEVC, VVC video encoder to classify the CU block type of the to-be-encoded video, and the specific process is as follows:

[0028] S51, according to the resolution of the current frame to be encoded, the number of 128*128 size image blocks of the frame is calculated. In the order of from left to right and from top to bottom, an image block is read into step S52; if the read image block is less than 128*128, step S59 is entered;

[0029] S52, the brightness information of each pixel of the 128*128 image block is extracted and saved; the brightness value of each pixel is normalized to a value in the interval of 0 to 1 to meet the requirements of neural network model input standardization, and step S53 is entered;

[0030] S53, the pre-trained CNN model is loaded, the normalized brightness information is converted into a tensor of model input format, and then input into the CNN model for inference, and step S54 is entered;

[0031] S54, the classification probability of each image block is generated after model inference, that is, three channel probability values corresponding to three different categories; the array is stored by using row-first Z-shaped indexing method, and the index of each image block is obtained by multiplying the row number of the image block by the width of the image block and adding the column number; the output result is stored in a specified memory for subsequent access and classification operation, and step S55 is entered;

[0032] S55, it is checked whether the current CU block to be encoded is a brightness component and the CU position is non-boundary, if yes, step S56 is entered; if not, step S59 is entered;

[0033] S56, each 4*4 pixel sub-region of the current CU block is traversed according to the classification probability output by the model; the classification probability of each sub-region is read in turn, the probability values of the three categories are compared, the category label with the highest probability is selected, and the label is assigned to the 4*4 region; in this way, the preliminary category distribution map of the current CU block is generated, and the category label of each 4*4 region is stored in a two-dimensional array for further statistical processing, and step S57 is entered;

[0034] S57, the final category label of the CU block is calculated by using bit operation: first, the category count array is initialized to count the number of each category in the current CU block. The category label of each region is counted into the corresponding category count, and the accumulation is performed class by class; the final category label variable of the CU block is initialized. Using bit operation combined with multiple category information, if a category appears in the CU block, the flag bit of the category is added to the category label of the CU block through bit operation, so that each bit represents a category. Through this way, when multiple categories appear, the category label will record all the category information; after this step, the category label of the CU block contains all the categories appearing in the current CU block, and step S58 is entered;

[0035] S58, parsing the content type according to the category label of the CU block. The category label of the CU block has represented all the category information in the CU block through bit operation, each bit corresponding to a category; by reading the state of each bit of the category label, the content type contained in the current CU block is determined, including the A category, the CC category, the TGM category and the Mixed category; the Mixed category here is a mixed type containing two or three of the A category, the CC category and the TGM category; the parsed category label provides content category support for the subsequent encoding process, so that the encoder can adaptively encode according to the characteristics of the CU block. Step S59 is entered;

[0036] S59, ending the current CU block type classification.

[0037] Further, in the step S57, the model outputs the category probability information of each CU block; after processing the output of the model, the probability information is used to calculate the occurrence frequency of each category in each 4*4 pixel unit; the CU block type contains four categories: the A category, the CC category, the TGM category and the Mixed category; according to the occurrence of the categories, the different categories are encoded using bit operation, and these information is recorded in the CU category code, so as to ensure that the category combination of the current CU block is efficiently stored and called.

[0038] Further, in the step S57, the calculation of the CU block category is realized through a bit operation formula, which is used to determine the combination of each category in the CU block; the CU block category contains four categories: the CC category, the A category, the TGM category and the Mixed category, and the following formula is used to calculate the CU block category code:

[0039]

[0040] CU Category Encoding represents the CU category; i represents the index value of the category, 0 represents the CC category, 1 represents the A category, 2 represents the TGM category, and others represent the Mixed category; P c represents a Boolean value, 1 if the category exists, otherwise 0; 1<<i represents that 1 is shifted to the position corresponding to the category by bit.

[0041] A storage medium, which internally stores a computer program, wherein the computer program is read by a processor to execute any one of the above-mentioned video encoding CU block classification methods based on a lightweight deep learning model.

[0042] The advantages and beneficial effects of the present application are as follows:

[0043] The application classifies the types of the current CU block in the coded frame by designing a lightweight convolutional neural network (CNN) model to distinguish different types of video content (natural content and screen content). First, the CNN model extracts and analyzes the features of the CU block to determine its type; for screen content (class A, TGM class, mixed class) and natural content (CC class), the model can effectively distinguish the texture and structural characteristics of different types of CU blocks. The classification result provides a basis for the subsequent encoder to select the most suitable CU block division mode and encoding tool, thereby realizing an adaptive encoding strategy.

[0044] To evaluate the encoding acceleration performance of the method proposed in the application, the application performs performance tests on the official encoder VTM-17.0, the test standard follows the common test condition (CTC) specified by the official standard dynamic range (SDR) video sequence, selects one sequence for each type of video for testing, and uses the all intra (AI) encoding configuration to perform encoding performance tests at four quantization parameter (QP) values {37, 32, 27, 22}. The BD-BR (bit-rate delta) and ΔT are used as performance evaluation indicators of the acceleration algorithm. BD-BR measures the loss of encoding efficiency, the larger the value, the more the loss of encoding efficiency; ΔT represents the time saved by the algorithm, and its calculation formula is:

[0045]

[0046] Where T anchor represents the encoding time of the original encoder, and T proposed represents the encoding time of the encoder using the model proposed in the application.

[0047] Table 1 shows the encoding performance of each type of video sequence. Proposed in Table 1 represents the method of the application, and Ref is the algorithm in the comparative literature (Wang D, Yu J, Lu X, et al. Fast Mode and CU Splitting Decision for Intra Prediction in VVC SCC [J]. IEEE Transactions on Broadcasting, 2024.). It can be seen that the method of the application can reduce the encoding time by an average of 54.66%, while the BD-BR increases by only 1.30%. From the comparison results, it can be seen that the performance of the model in this paper is significantly better than that of the algorithm in the comparative literature.

[0048] ​Table 1 Performance comparison of the present application and the reference method

[0049]

[0050] Figure 1 The accuracy and loss curves in the model training process are shown. During the model training process, the accuracy and loss curves both show good convergence and stability. The accuracy curve shows that the accuracy of the training set and the test set gradually increases with the increase of the training period, rapidly increases within the first 100 epochs, and finally stabilizes above 90%, indicating that the model has good classification performance and no obvious overfitting phenomenon. The loss curve rapidly decreases in the early stage of training, verifying that the model effectively minimizes the error, and tends to be stable after about 600 epochs, showing that the model has approached the optimal state. The loss curves of the training and test sets are close to each other, further proving that the model has good generalization ability and can efficiently complete the CU block type classification task.

[0051] The innovation of the present application mainly lies in claim 4 and claim 6. The method of claim 4 provides high-quality training data for the CNN model through data preprocessing and label annotation. The method of claim 6 classifies the CU block type using the trained lightweight CNN model. The present application selects different encoding modes for subsequent CU of different types under the condition that the video quality is basically unchanged and the encoding output bit rate increases little, effectively reducing the computational burden of the VVC encoder in complex content processing. By optimizing the recognition process of the CU type, the present application is applicable to high-resolution video scenarios such as high-definition, ultra-high-definition video and screen content encoding, and can improve the encoding efficiency and shorten the encoding time under the premise of meeting the real-time requirement. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is the accuracy and loss curve in the model training process provided by the preferred embodiment of the present application.

[0053] Figure 2 is a flowchart of a video encoding CU block classification method based on a lightweight deep learning model provided by the preferred embodiment of the present application.

[0054] Figure 3 is the structure of the lightweight CNN model. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments of the present application.

[0056] The technical solution of the present application to solve the above technical problems is:

[0057] Figure 2 For the video coding CU block classification method based on a lightweight deep learning model in the application, the method comprises the following steps:

[0058] S1, a lightweight convolutional neural network (CNN) model is designed for coding unit (CU) type classification.

[0059] S2, a video sequence dataset for training the CNN model is collected, the video sequence covers A (Animation), CC (Camera-captured Content) and TGM (Text and Graphics with Motion). The collected dataset is divided, 20% of the samples are used as the test set, and 80% of the samples are used as the training set.

[0060] S3, the collected video sequence set is preprocessed. Different types of video data are labeled to build a training dataset for training the CNN model.

[0061] S4, based on the training dataset built in step S3, the designed CNN model is trained using machine learning method, the model parameters are updated using back propagation algorithm, the performance of the trained model is evaluated using the validation set, and the accuracy of the model is optimized.

[0062] S5, the trained CNN model in step S4 is deployed to a high efficiency video coding (HEVC) video encoder, a general video coding (VVC) video encoder, etc. The CU type of the video to be coded is classified, which is used for subsequent CU block division and other processing processes to improve the coding efficiency.

[0063] Preferably, in the step S1, a lightweight convolutional neural network (CNN) model is designed for coding unit (CU) type classification, specifically including: the lightweight CNN model adopts a symmetric encoder-decoder structure. The encoder gradually extracts features through 6 convolutional layers, batch normalization layers, and ReLU activation functions, and reduces the input 128x128 single-channel image to 2x2x512; the bottleneck layer further compresses the feature map to 1x1x1024 to extract deep-level features; the decoder gradually restores the size of the feature map through 5 deconvolutional layers, and outputs a 32x32x3 classification result. The model does not use skip connections to avoid additional channel growth, reduce the number of parameters, and make the model layer simple and efficient; finally, the output layer uses 1x1 convolution and Softmax function to classify each pixel and complete the classification of CU type.

[0064] Preferably, in the step S2, a video sequence dataset for training the CNN model is collected, and the video sequence covers A, CC, and TGM classes. The A video sequence is a computer-generated animation video; the CC video sequence is a video shot by a camera; and the TGM video sequence is a video with moving text and graphics. The collected video sequences are mainly from the Internet and the test sequence set used by video coding standards.

[0065] Preferably, in the step S3, the collected video sequence set is preprocessed. Different types of video data are labeled. The collected video sequences are preprocessed through platforms such as Python, including grayscale, normalization, and cropping, specifically including the following steps: first, the selected video sequence is subjected to grayscale processing to convert a color image into a single-channel grayscale image, thereby simplifying the data and improving the processing efficiency; second, each frame of image is cropped to an image block of 128x128 pixels to ensure the standardization of input data; then, the cropped image block is labeled as CC, A, and TGM. The preprocessing of this step provides standardized data input for the training of the CNN model.

[0066] Preferably, in the step S4, the designed CNN model is trained using a machine learning method, and the model parameters are updated using a backpropagation algorithm. The loss function used for training the designed CNN model is:

[0067]

[0068] where Loss is the loss; C is the total number of classes; y i is the true label; p i is the probability predicted by the model that belongs to class i.

[0069] In the backpropagation algorithm, the gradient calculation formula for each layer parameter θ l is:

[0070]

[0071] wherein a l+1 is the activation output of the l+1th layer; z l represents the linear transformation output of the lth layer; θ l represents the parameters of the lth layer.

[0072] The parameter update of each layer uses the Adam optimizer to update the parameters, and the update formula is:

[0073]

[0074] wherein η is the learning rate.

[0075] Preferably, in the step S5, the trained CNN model is deployed into a video encoder such as HEVC, VVC, etc. to classify the CU block type of the video to be encoded, and the specific process is as follows:

[0076] S51, according to the resolution of the current frame to be encoded, the number of 128x128 size image blocks of the frame is calculated. Read an image block in the order from left to right and from top to bottom, and enter step S52; if the read image block is less than 128x128, enter step S59.

[0077] S52, extract and save the brightness information of each pixel of the 128x128 image block. Normalize the brightness value of each pixel to a value in the interval of 0 to 1 to meet the standardization requirements of the neural network model input, and enter step S53.

[0078] S53, load the pre-trained CNN model. Convert the normalized brightness information into a tensor (Tensor) in the input format of the model, and then input it into the CNN model for inference, and enter step S54.

[0079] S54, generate the classification probability of each image block after model inference, which is the probability value of three channels corresponding to three different categories. Store the array using the row-first Z-shaped indexing method, and the index of each image block is obtained by multiplying the row number of the image block by the width of the image block and adding the column number. Store the output result in the specified memory for subsequent access and classification operation, and enter step S55.

[0080] S55, check whether the current CU block to be encoded is a luminance component and the CU position is non-boundary, if yes, enter step S56; if not, enter step S59.

[0081] S56, the classification probability output by the model traverses each 4x4 pixel sub-region of the CU block. The classification probability of each sub-region is read in turn, and the probability values of the three categories are compared to select the category label with the highest probability, and the label is assigned to the 4x4 region. In this way, a preliminary category distribution map of the current CU block is generated, and the category label of each 4x4 region is stored in a two-dimensional array for further statistical processing, entering step S57.

[0082] S57, the final category label of the CU block is calculated using bit operations: first, initialize the category count array to count the number of each category in the current CU block. Traverse all 4x4 sub-regions in the CU block, and add the category label of each region to the corresponding category count, and accumulate by category; initialize the final category label variable of the CU block. Use bit operations to combine multiple category information. If a category appears in the CU block, add the flag bit of the category to the category label of the CU block through bit operations, so that each bit represents a category. In this way, when multiple categories appear, the category label will record all the category information that appears. After this step, the category label of the CU block contains all the categories that appear in the current CU block, and enters step S58.

[0083] S58, the content type of the CU block is parsed according to the category label of the CU block. The category label of the CU block has been represented by bit operations to indicate all the category information in the CU block, and each bit corresponds to a category. By reading the state of each bit of the category label, the content type contained in the current CU block is determined, including A, CC, TGM and Mixed. The Mixed here is a mixed type containing two or three of A, CC and TGM. The parsed category label provides content category support for the subsequent encoding process, so that the encoder can adaptively encode according to the characteristics of the CU block. Enter step S59.

[0084] S59, end the current CU block type classification.

[0085] Preferably, in step S57, the model outputs category probability information for each CU block. After processing the output of the model, the probability information is used to calculate the frequency of each category in each 4x4 pixel unit. The CU block type contains four categories: A, CC, TGM and Mixed. According to the occurrence of the categories, different categories are encoded using bit operations, and these information is recorded in the CU category code to ensure that the category combination of the current CU block is efficiently stored and called.

[0086] The step S57, the calculation of the CU block class is realized by a bitwise operation formula, which is used to determine the combination of each class in the CU block. The CU block class includes four classes: CC class, A class, TGM class and Mixed class. The following formula is used to calculate the CU block class code:

[0087]

[0088] wherein, CU Category Encoding represents the CU class; i represents the index value of the class (0 represents the CC class, 1 represents the A class, 2 represents the TGM class, and others represent the Mixed class); P c represents a Boolean value, 1 if the class exists, otherwise 0; 1<<i represents that 1 is shifted to the left to the position corresponding to the class.

[0089] A storage medium internally stores a computer program, which is read by a processor to execute the video coding CU block classification method based on the lightweight deep learning model according to any one of the above embodiments.

[0090] The system, device, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions.

[0091] The computer readable medium includes permanent and non-permanent, removable and non-removable media, which can be realized by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition in this paper, computer readable medium does not include transitory computer readable medium, such as modulated data signal and carrier wave.

[0092] It is also to be noted that the terms "comprising", "including", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0093] The above examples are to be understood only as illustrative of the application and not a restriction on the scope of protection of the application. After reading the specification, the skilled person can make various changes or modifications to the application, and these equivalent changes and modifications also fall within the scope defined by the claims of the application.

Claims

1. A method for CU block classification in video coding based on lightweight deep learning model, characterized in that, The method comprises the following steps: S1, a lightweight convolutional neural network (CNN) model is designed for classification of coding unit (CU) types; S2, a video sequence dataset for training the CNN model is collected, the video sequence covers A animation, CC camera shooting content, and TGM text and graphics with motion, the collected dataset is divided, 20% of the samples are used as a test set, and 80% of the samples are used as a training set; S3, the collected video sequence set is preprocessed, different types of video data are labeled, and a training dataset for training the CNN model is constructed; S4, based on the training dataset constructed in step S3, the designed CNN model is trained using a machine learning method, the model parameters are updated using a back propagation algorithm, the performance of the trained model is evaluated using a verification set, and the accuracy of the model is optimized; S5, the trained CNN model in step S4 is deployed into a high efficiency video coding (HEVC) and a versatile video coding (VVC) video encoder to classify the CU types of a video to be encoded; In step S5, the trained CNN model is deployed into the HEVC and VVC video encoders to classify the CU block types of a video to be encoded, and the specific process is as follows: S51, the number of 128*128 image blocks of a current frame to be encoded is calculated according to the resolution of the frame, an image block is read in the order from left to right and from top to bottom, and step S52 is entered; if the read image block is less than 128*128, step S59 is entered; S52, the brightness information of each pixel of the 128*128 image block is extracted and saved; the brightness value of each pixel is normalized to a value in the interval of 0 to 1 to meet the standardization requirements of the neural network model input, and step S53 is entered; S53, the pre-trained CNN model is loaded, the normalized brightness information is converted into a tensor of the model input format, and then input into the CNN model for inference, and step S54 is entered; S54, the classification probability of each image block is generated after model inference, that is, three probability values of three channels respectively corresponding to three different categories; the array is stored by using a row-first Z-shaped indexing method, and the index of each image block is obtained by multiplying the row number of the image block by the width of the image block and adding the column number; The output result is stored in a specified memory for subsequent access and classification operation, and step S55 is entered; S55, whether the current CU block to be encoded is a brightness component and the CU position is a non-boundary is checked, if yes, step S56 is entered; if not, step S59 is entered; S56, each 4*4 pixel sub-region of the CU block is traversed according to the classification probability output by the model; the classification probability of each sub-region is read in turn, the probability values of the three categories are compared, the category label with the highest probability is selected, and the label is assigned to the 4*4 region; in this way, a preliminary category distribution map of the current CU block is generated, and the category label of each 4*4 region is stored in a two-dimensional array for further statistical processing, and step S57 is entered; S57, calculating the final category label of the CU block by using bit operation: first, initializing the category count array to count the number of each category in the current CU block; traversing all 4x4 sub-regions in the CU block, counting the category label of each region into the corresponding category count, and accumulating by category; initializing the final category label variable of the CU block; using bit operation to combine multiple category information, if a category appears in the CU block, the flag bit of the category is added to the category label of the CU block through bit operation, so that each bit represents a category; in this way, when multiple categories appear, the category label will record all the category information; after this step, the category label of the CU block contains all the categories appearing in the current CU block, and step S58 is entered; S58, parsing the content type according to the category label of the CU block; the category label of the CU block has been represented by bit operation, which contains all the category information in the CU block, and each bit corresponds to a category; by reading the state of each bit of the category label, the content type contained in the current CU block is determined, including A, CC, TGM and Mixed; the Mixed here is a mixed type containing two or three of A, CC and TGM; the parsed category label provides content category support for the subsequent encoding process, so that the encoder can adaptively encode according to the characteristics of the CU block; entering step S59; S59, ending the current CU block type classification; In step S57, the model output is the category probability information of each CU block; after processing the output of the model, the probability information is used to calculate the frequency of each category appearing in each 4x4 pixel unit; the CU block type contains four categories: A, CC, TGM and Mixed; according to the occurrence of the category, the different categories are encoded by using bit operation, and these information is recorded in the CU category code, so as to ensure that the category combination of the current CU block is efficiently stored and called; In step S57, the calculation of the CU block category is realized by using the bit operation formula to determine the combination of each category in the CU block; the CU block category contains four categories: CC, A, TGM and Mixed, and the following formula is used to calculate the CU block category code: wherein CU Category Encoding represents the CU category; i represents the index value of the category, 0 represents the CC category, 1 represents the A category, 2 represents the TGM category, and others represent the Mixed category; P c represents a Boolean value, 1 if the category exists, and 0 otherwise; 1 << i represents shifting 1 to the left by the position corresponding to the category.

2. The method of claim 1, wherein the method comprises: In step S1, a lightweight convolutional neural network (CNN) model is designed for the classification of coding unit (CU) type, which specifically includes: the lightweight CNN model adopts a symmetric encoder-decoder structure, the encoder of which gradually extracts features through 6 convolutional layers, batch normalization layers and ReLU activation functions, and reduces the input 128x128 single-channel image to 2x2x512; the bottleneck layer further compresses the feature map to 1x1x1024 to extract deep-level features; the decoder gradually restores the size of the feature map through 5 deconvolutional layers, and outputs a 32x32x3 classification result; the model does not use skip connection to avoid additional channel growth; finally, the output layer uses 1x1 convolution and Softmax function to classify each pixel, completing the classification of the CU type. 3.The method of claim 1, wherein, In the step S2, a video sequence dataset for training the CNN model is collected, the video sequence covers A, CC and TGM, the A video sequence is a computer-generated animation video, the CC video sequence is a video shot by a camera, and the TGM video sequence is a video with moving text and graphics, and the collected video sequence is mainly from the Internet and a test sequence set used by a video coding standard.

4. The method of claim 1, wherein the method is based on a lightweight deep learning model. In the step S3, the collected video sequence set is preprocessed, and different types of video data are labeled, the collected video sequence is preprocessed by grayscale, normalization and cropping through a Python platform, and the preprocessing specifically includes the following steps: first, the selected video sequence is subjected to grayscale processing, and a color image is converted into a single-channel grayscale image to simplify data and improve processing efficiency; second, each frame of image is cropped into an image block with a size of 128*128 pixels to ensure the standardization of input data; then, the cropped image block is subjected to content labeling and is divided into CC, A and TGM; the preprocessing of this step provides standardized data input for the training of the CNN model.

5. The method of claim 1, wherein the method is based on a lightweight deep learning model. In the step S4, the designed CNN model is trained by using a machine learning method, a back propagation algorithm is used to update model parameters, and a loss function used for training the designed CNN model is as follows: wherein Loss is a loss; C is a total number of classes; y i is a true label; p i is a probability predicted by the model that belongs to class i; In the backpropagation algorithm, the gradient of the formula for calculating each layer of parameters θ l is: wherein a l+1 is the activation output of the l+1th layer; z l represents the linear transformation output of the lth layer, θ l represents the parameters of the lth layer; The parameter update of each layer uses an Adam optimizer to update parameters, and an update formula is as follows: Wherein, η is a learning rate.

6. A storage medium, which stores a computer program in the inside, characterized by The computer program is read by a processor, and the video coding CU block classification method based on the lightweight deep learning model in any one of claims 1-5 is executed.

Images