HEVC (High Efficiency Video Coding) intra-frame CU (Coding Unit) division method and device based on deep learning and coding method

Abstract

The invention discloses an HEVC (High Efficiency Video Coding) intra-frame CU (Coding Unit) division method and device based on deep learning and a coding method. The method comprises the following steps: firstly, acquiring video or image information of a coding unit to be divided; carrying out standardization processing on the video or image information to obtain a CTU image to be coded; then, the CTU image is input into a CPAC-I model, a CU division decision is obtained, and the CPAC-I model is a neural network model based on combination of an attention mechanism of SKnet and a convolution technology; and finally, according to the CU division decision, dividing the HEVC frame so as to complete the HEVC intra-frame CU division based on deep learning. The method can effectively improve the overall performance of video coding and decoding, and can flexibly adapt to different types of video contents and coding scenes.

Description

Technical Field

[0001] The present invention relates to a communication network, and in particular to a HEVC intra-frame CU division method, device and encoding method based on deep learning. Background Art

[0002] With the substantial increase in the coverage and speed of broadband Internet services, the share of video in global data traffic has reached about 80%, and this figure continues to grow. In order to efficiently store and transmit video, the research on video coding technology is particularly important.

[0003] In 2013, the International Telecommunications union for Telecommunications Standardization Sector (ITU-T) and the International Standardization Organization / International Electrotechnical Commission (ISO / IEC) jointly launched the new generation of video coding standard H.265 / High Efficiency Video Coding (HEVC). Compared with its predecessor H.264 / Advanced Video Coding (AVC), the HEVC standard saves about 50% of the bit rate at similar video quality.

[0004] This is due to the HEVC standard's use of more advanced coding and decoding algorithm technology. In the intra-frame prediction mode of the predictive coding stage, in order to better eliminate the spatial redundancy in a frame of image, each frame of image is first divided into blocks of different sizes based on the correlation between adjacent pixels, and intra-frame coding is performed in blocks. For each frame of image, the rate distortion optimization (RDO) criterion is used to determine whether a frame of image needs to be divided into blocks, and how large the blocks should be to minimize spatial redundancy, which greatly increases the encoding time.

[0005] In essence, video coding uses the high texture correlation of images within frames in natural video signals. In recent years, deep neural networks, due to their powerful nonlinear feature expression capabilities, can effectively learn visual features from the bottom to the top from big data, and have a complete set of fairly mature training and testing systems. They have achieved great success in the field of computer vision in low-level tasks such as denoising and super-resolution, as well as high-level semantic tasks such as classification and detection.

[0006] High Efficiency Video Coding (HEVC) has greatly reduced the bit rate of the previous H.264 standard, but at the cost of extremely high coding complexity. In HEVC, the quadtree division of coding units (Code Unit, CU) takes up 80% of the coding time due to the brute force search for rate-distortion optimization, consuming a large part of the HEVC coding complexity.

[0007] In order to improve the performance of video encoding and decoding, researchers need to find more effective features as input, which is usually quite difficult. On the other hand, each CU depth usually requires one or more classifiers, resulting in greater workload and longer training time. The existing technology improves the CU (Coding Unit) division method through the intra-frame block division RDO decision algorithm to improve the encoding and decoding efficiency. However, the RDO algorithm usually only considers the coding optimization of each block, lacks a global perspective and information interaction, and is prone to fall into a local optimal solution and cannot achieve the global optimal coding effect. In addition, the RDO algorithm often relies on heuristic rules to assist in decision-making. The design of rules and the selection of parameters require experience and tuning, which may not necessarily achieve the best performance. In addition, the design of the RDO algorithm is relatively fixed and cannot flexibly adapt to different types of video content and encoding scenarios. Summary of the invention

[0008] The technical problem to be solved by the present invention is to propose a HEVC intra-frame CU division method, device and encoding method based on deep learning in view of the above-mentioned shortcomings of the prior art. The method can effectively improve the global performance of video encoding and decoding, and can flexibly adapt to different types of video content and encoding scenarios.

[0009] In a first aspect, the present invention provides a HEVC intra-frame CU division method based on deep learning, the method comprising the following steps:

[0010] Step S1: Obtaining video or image information to be divided into coding units;

[0011] Step S2: normalizing the video or image information to obtain a CTU image to be encoded;

[0012] Step S3: inputting the CTU image into the CPAC-I model to obtain a CU partition decision;

[0013] The CPAC-I model is a neural network model based on the SKnet attention mechanism and convolution technology.

[0014] Step S4: Divide the HEVC frame according to the CU division decision to complete the deep learning-based intra-HEVC frame CU division.

[0015] Further, before the step S3, the method includes step S0;

[0016] Step S0: constructing a CPAC-I model;

[0017] The step S0 specifically includes the following steps:

[0018] Step S01: creating a data set based on historical video or image information, and dividing the data set into a training set, a test set, and a validation set; and designing a loss function;

[0019] Step S02: Based on the loss function, the CU partitioning CPAC-I network is trained according to the training set; and the effect of the trained CU partitioning CPAC-I network is tested according to the test set; and the parameters in the trained CU partitioning CPAC-I network are adjusted according to the validation set, so as to obtain a CPAC-I model;

[0020] Among them, the CU partition CPAC-I network is a pre-built neural network, and the CU partition CPAC-I network includes a feature extraction unit and a split decision unit. The feature extraction unit is used to learn and extract the key features of the image, and the split decision unit is used to make decisions on partitioning based on the key features of the image extracted through learning; the key features of the image are image features that affect the CTU image partition decision or the coding structure.

[0021] Furthermore, before step S02, the method further includes step A:

[0022] Step A: Construct CU partition CPAC-I network;

[0023] The step A specifically comprises the following steps:

[0024] Step A1: Take the brightness component L as the input of the neural network and input it into the three parallel branches of the neural network to obtain the first branch image feature quantity The second branch image feature The third branch image feature

[0025] Step A2: Fuse the first branch image feature quantity, the second branch image feature quantity, and the third branch image feature quantity to obtain the total unfiltered fusion feature quantity G s ;

[0026]

[0027] in,

[0028] represents the first branch image feature, represents the second branch image feature, represents the third branch image feature;

[0029] Step A3: According to the total amount of unscreened fusion features G s , based on the attention weights of different kernels, the final feature map V of the three branches is obtained (1) , V (2) and V (3) ;

[0030] Step A4: According to the final feature map V of the three branches (1) , V (2) and V (3) , the split decision is obtained to perform CU partitioning, thereby constructing a CU partitioning CPAC-I network.

[0031] Furthermore, the step A3 specifically includes the following steps:

[0032] Step A31: According to the total amount of unscreened fusion features G s , generate the characteristic quantity s of the cth element in the channel statistics vector s c ;

[0033]

[0034] Among them, G s (i,j) represents the total amount of unfiltered fusion features in the (i,j)th position;

[0035] H and W represent the height and width of the feature map, respectively;

[0036] Step A32: Set the feature quantity s c Send it to the fully connected layer to get a compact feature vector z;

[0037] z=δ(B(Ws c ));

[0038] Where δ represents the ReLU function, B() function represents batch normalization, and weight

[0039] The formula for determining the value of d is as follows:

[0040]

[0041] Where r represents the reduction rate;

[0042] d L represents the minimum value of d;

[0043] Step A33: Apply the softmax operator to the feature vector z to obtain the soft attention vector

[0044]

[0045] in, u 1 ,u 2 ,u 3 Respectively represent U 1 ,U 2 ,U 3 The soft attention vector of represents the cth row of A, represents the cth row of B, represents the cth row of D, Respectively represent u 1 ,u 2 ,u 3 The cth element in ;

[0046] Step A34: According to the soft attention vector Get the final feature map V of the three branches (1) , V (2) and V (3) ;

[0047]

[0048] The l-th layer feature map

[0049] Furthermore, in step S01, the step of preparing a data set based on historical video or image information specifically includes the following steps:

[0050] Step B1: randomly shuffle the image frames in the historical video or image information, and then divide them into a training set, a test set, and a validation set in a ratio of 4:1:1;

[0051] Step B2: splice and convert the images in the divided training set, test set and validation set into YUV format data subsets with different resolutions;

[0052] Step B3: label the YUV format data subset using HEVC video encoding to obtain a data set with network training labels, thereby completing the production of the data set;

[0053] Among them, the network training labels have three layers: the first layer is the network training label of whether 64x64CU can be divided into 32x32CU; the second layer is the network training label of whether 32x32CU can be divided into 16x16CU; the third layer is the network training label of whether 16x16CU can be divided into 8x8CU.

[0054] Furthermore, in step B3, the network training label is the real label of the data set under different quantization parameters; wherein the quantization parameters include four quantization levels, namely 22, 27, 32 and 37;

[0055] The normalization processing formula of the quantization parameter is as follows:

[0056]

[0057] Among them, q is the original quantization parameter QP value, is the normalized quantization parameter QP value.

[0058] Furthermore, in step S01, the design loss function includes the following steps:

[0059] Step C1: According to the loss of each branch, calculate the sum of the losses of each branch Loss sum ;

[0060] Among them, the loss formula of the λth branch is as follows:

[0061]

[0062] Loss (λ) is the loss of the λth layer, χ is the normalization constant, y (λ) (G i,j ) represents the true value label of the λth layer, represents the predicted value of the λth layer, H(·) represents the cross entropy loss function, and H and W represent the height and width of the feature map respectively;

[0063] The sum of the losses of each branch Loss sum The calculation formula is as follows:

[0064]

[0065] Step C2: Calculate the cross entropy loss function for each layer based on the RD cost loss function The specific formula is as follows:

[0066]

[0067] is the normalized RD cost;

[0068] ζ represents the partitioning mode;

[0069] r ξ Indicates the RD cost of the nth CU of the current CTU under the partition mode ζ;

[0070] r min represents the minimum RD cost of CU in all possible partitioning modes;

[0071] Step C3: According to Loss sum and Get the overall loss function of the neural network, thereby completing the design of the loss function;

[0072]

[0073] Among them, L CPAC-I is the overall design function of the neural network.

[0074] In a second aspect, the present invention provides a HEVC video intra-frame coding method, the method comprising the following steps:

[0075] Step K1: using the HEVC intra-frame CU division method based on deep learning described in the first aspect to divide the HEVC frame;

[0076] Step K2: transform, quantize and entropy encode the divided HEVC frames to obtain encoding preprocessing data;

[0077] Step K3: Pack the encoding preprocessing data to obtain a video encoding stream, thereby completing HEVC video intra-frame encoding.

[0078] In a third aspect, the present invention provides a HEVC intra-frame CU division device based on deep learning, the device comprising:

[0079] An acquisition unit, used to acquire video or image information to be divided into coding units;

[0080] A first processing unit, connected to the acquisition unit, configured to perform standardization processing on the video or image information to obtain a CTU image to be encoded;

[0081] A second processing unit, connected to the first processing unit, configured to input the CTU image into a CPAC-I model to obtain a CU partition decision;

[0082] The CPAC-I model is a neural network model based on the SKnet attention mechanism and convolution technology.

[0083] The third processing unit is connected to the second processing unit, and is used to divide the HEVC frame according to the CU division decision to complete the deep learning-based intra-HEVC frame CU division.

[0084] Furthermore, the device also includes:

[0085] Building blocks for constructing the CPAC-I model;

[0086] The building block comprises:

[0087] A processing module, for preparing a data set based on historical video or image information, and for dividing the data set into a training set, a test set, and a validation set; and for designing a loss function;

[0088] a design module connected to the processing module, for training the CU partitioning CPAC-I network according to the training set based on the loss function; and testing the effect of the trained CU partitioning CPAC-I network according to the test set; and adjusting the parameters in the trained CU partitioning CPAC-I network according to the validation set, thereby obtaining a CPAC-I model;

[0089] Among them, the CU partition CPAC-I network is a pre-built neural network, and the CU partition CPAC-I network includes a feature extraction unit and a split decision unit. The feature extraction unit is used to learn and extract the key features of the image, and the split decision unit is used to make decisions on partitioning based on the key features of the image extracted through learning; the key features of the image are image features that affect the CTU image partition decision or the coding structure.

[0090] Beneficial effects of the present invention:

[0091] 1. Improve the global performance of video encoding and decoding

[0092] The CU partitioning method of the present invention can better consider global information, achieve global optimization, and improve the global performance of video encoding and decoding by integrating the attention mechanism. This global perspective helps to make more reasonable decisions between different areas of the video frame, improve the overall coding efficiency and quality, and avoid local optimal problems.

[0093] 2. Higher flexibility and stronger adaptability

[0094] The deep learning model CPAC-I model of the present invention has high flexibility and adaptability. After being trained with a large amount of video data, it can be dynamically adjusted according to specific circumstances to ensure the encoding effect.

[0095] 3. Improve the efficiency of feature learning

[0096] The present invention is based on the CPAC-I model, which can efficiently learn important features from the input CTU image by combining the attention mechanism and convolution technology of SKnet. This feature learning is automatic and dynamic, and can adaptively adjust the feature extraction process according to different video content, greatly improving the ability to understand complex video content, thereby optimizing CU division decisions.

[0097] 4. Improve computing efficiency

[0098] The present invention can quickly divide CTU into CU, thereby improving encoding speed and real-time performance.

[0099] 5. Reduce manual intervention

[0100] The present invention can automatically learn and optimize these rules and parameters through large-scale data training, reducing the reliance on human intervention. This not only speeds up the deployment and application of the algorithm, but also enables the model to continuously improve and optimize itself through new data.

[0101] 6. Improve coding quality

[0102] Through efficient feature learning and global optimization, the present invention can more accurately capture the details and structural information of video content, thereby optimizing encoding decisions. This optimization not only improves compression efficiency, but also improves the subjective and objective quality of the video at the same bit rate, providing users with a better viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] Figure 1 Schematic diagram of a HEVC intra-frame CU division method based on deep learning in an embodiment of the present invention;

[0104] Figure 2 Schematic diagram of a HEVC intra-frame CU division device based on deep learning in an embodiment of the present invention;

[0105] Among them, the figure numbers are: 10, acquisition unit, 20, first processing unit, 30, second processing unit, 40, third processing unit. DETAILED DESCRIPTION

[0106] In order to enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0107] It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention rather than to limit the present invention.

[0108] It can be understood that, in the absence of conflict, the various embodiments of the present invention and the various features in the embodiments can be combined with each other.

[0109] It can be understood that, for the convenience of description, the drawings of the present invention only show the parts related to the present invention, while the parts irrelevant to the present invention are not shown in the drawings.

[0110] It can be understood that each unit and module involved in the embodiments of the present invention may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.

[0111] It can be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present invention may occur in an order different from that marked in the drawings.

[0112] It is understood that the flowcharts and block diagrams of the present invention illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present invention. Each box in the flowchart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or may be implemented by a combination of hardware and computer instructions.

[0113] It can be understood that the units and modules involved in the embodiments of the present invention can be implemented by software or hardware. For example, the units and modules can be located in a processor.

[0114] Embodiment 1:

[0115] like Figure 1 As shown, this embodiment provides a HEVC intra-frame CU division method based on deep learning, and the method includes the following steps:

[0116] Step S1: Obtain video or image information to be divided into coding units.

[0117] The coding unit to be divided is an important concept in video coding. In video coding, a video frame is divided into multiple macroblocks or blocks, each of which is called a "coding unit". The "coding unit to be divided" refers to the part of the video frame to be encoded and compressed that needs to be divided. This subdivision is to improve the efficiency of encoding and video quality. For image information, a similar division and encoding process can also be performed. In image compression and processing, the image is also divided into several coding units for compression and transmission.

[0118] Step S2: Standardize the video or image information to obtain a CTU image to be encoded.

[0119] In video coding, CTU (Coding Tree Unit) refers to the coding tree unit, which is the basic unit for encoding and compression in the video frame. Standardizing the video to obtain the CTU image to be encoded means performing a series of standardization processes on the video image in order to divide it into CTUs so that each CTU can be independently encoded and compressed.

[0120] The normalization process may include one or more of the following steps:

[0121] (1) Resolution adjustment: Adjust the resolution of a video or image to a standard size for subsequent processing.

[0122] (2) Color space conversion: Convert the image from RGB color space to YUV color space because YUV color space is more suitable for video compression encoding and the encoder processes brightness information (Y) and chrominance information (U, V) more efficiently.

[0123] (3) Inter-frame prediction: Perform inter-frame prediction on the video, compare the current frame with the previous frame, and extract motion information for subsequent difference encoding.

[0124] (4) CTU partitioning: The processed video or image is divided into CTUs, usually in the form of the largest coding tree block, to facilitate subsequent coding unit extraction and processing.

[0125] Finally, the obtained CTU image to be encoded is the video or image that has undergone the above standardization processing, in which each CTU has been prepared for video encoding and compression.

[0126] Step S3: inputting the CTU image into the CPAC-I model to obtain a CU partition decision;

[0127] Among them, the CPAC-I model is a neural network model that combines the attention mechanism and convolution technology based on SKnet. The CPAC-I model is a neural network model, and its name CPAC-I stands for "CU partition network based on attention and convolution". In this model, the attention mechanism and convolution technology are combined. The attention mechanism allows the network to "focus" on specific parts of the input data for more efficient learning and inference. Convolution technology can capture spatial features in the input data and reduce the number of parameters, thereby improving the efficiency and generalization ability of the model. By combining these two technologies together, the CPAC-I model can better handle complex data and improve the performance and efficiency of the model.

[0128] Step S4: Divide the HEVC frame according to the CU division decision to complete the deep learning-based intra-HEVC frame CU division.

[0129] As a specific implementation, before step S3, the method includes step S0;

[0130] Step S0: constructing a CPAC-I model;

[0131] The step S0 specifically includes the following steps:

[0132] Step S01: creating a data set based on historical video or image information, and dividing the data set into a training set, a test set, and a validation set; and designing a loss function;

[0133] Step S02: Based on the loss function, the CU partitioning CPAC-I network is trained according to the training set; and the effect of the trained CU partitioning CPAC-I network is tested according to the test set; and the parameters in the trained CU partitioning CPAC-I network are adjusted according to the validation set, so as to obtain a CPAC-I model;

[0134] Among them, the CU partition CPAC-I network is a pre-built neural network, and the CU partition CPAC-I network includes a feature extraction unit and a split decision unit. The feature extraction unit is used to learn and extract the key features of the image, and the split decision unit is used to make decisions on partitioning based on the key features of the image extracted through learning; the key features of the image are image features that affect the CTU image partition decision or the coding structure.

[0135] As a specific implementation, before step S02, the method further includes step A:

[0136] Step A: Construct CU partition CPAC-I network;

[0137] The step A specifically comprises the following steps:

[0138] Step A1: Take the brightness component L as the input of the neural network and input it into the three parallel branches of the neural network to obtain the first branch image feature quantity The second branch image feature The third branch image feature

[0139] Step A2: Fuse the first branch image feature quantity, the second branch image feature quantity, and the third branch image feature quantity to obtain the total unfiltered fusion feature quantity G s ;

[0140]

[0141] in,

[0142] represents the first branch image feature, represents the second branch image feature, represents the third branch image feature;

[0143] Step A3: According to the total amount of unscreened fusion features G s , based on the attention weights of different kernels, the final feature map V of the three branches is obtained(1) , V (2) and V (3) ;

[0144] Step A4: According to the final feature map V of the three branches (1) , V (2) and V (3) , the split decision is obtained to perform CU partitioning, thereby constructing a CU partitioning CPAC-I network.

[0145] As a specific implementation, the step A3 specifically includes the following steps:

[0146] Step A31: According to the total amount of unscreened fusion features G s , generate the characteristic quantity s of the cth element in the channel statistics vector s c ;

[0147]

[0148] Among them, G s (i,j) represents the total amount of unfiltered fusion features in the (i,j)th position;

[0149] H and W represent the height and width of the feature map, respectively;

[0150] Step A32: Set the feature quantity s c Send it to the fully connected layer to get a compact feature vector z;

[0151] z=δ(B(Ws c ));

[0152] Where δ represents the ReLU function, B() function represents batch normalization, and weight

[0153] The formula for determining the value of d is as follows:

[0154]

[0155] Where r represents the reduction rate;

[0156] d L represents the minimum value of d;

[0157] Step A33: Apply the softmax operator to the feature vector z to obtain the soft attention vector

[0158]

[0159] Soft attention vector refers to a technique used in the attention mechanism to achieve flexible weighting of input information in neural networks. Soft attention vector allows the neural network to dynamically focus on different parts of the input and learn the weights of each part in order to effectively process the input data.

[0160] in, u 1 ,u 2 ,u 3 Respectively represent U 1 ,U 2 ,U 3 The soft attention vector of represents the cth row of A, represents the cth row of B, represents the cth row of D, Respectively represent u 1 ,u 2 ,u 3 The cth element in ;

[0161] Step A34: According to the soft attention vector Get the final feature map V of the three branches (1) , V (2) and V (3) ;

[0162]

[0163] The l-th layer feature map

[0164] As a specific implementation, in step S01, the step of making a data set based on historical video or image information specifically includes the following steps:

[0165] Step B1: Randomly shuffle the image frames in the historical video or image information, and then divide them into a training set, a test set, and a validation set in a ratio of 4:1:1.

[0166] The shuffled image frames are divided into a training set, a test set, and a validation set according to a ratio of 4:1:1. This embodiment adopts a ratio of 4:1:1, but other ratios may be adopted according to specific circumstances. If 4:1:1 is adopted, then:

[0167] (1) Training set: 80% of the total number of image frames, used for model training.

[0168] (2) Test set: accounts for 10% of the total number of image frames and is used for preliminary verification and evaluation of the results after model training.

[0169] (3) Validation set: accounts for 10% of the total number of image frames and is used to verify and adjust the model during the training process.

[0170] Step B2: Splice and convert the images in the divided training set, test set, and validation set into YUV format data subsets with different resolutions.

[0171] YUV is a color coding system that separates luminance and chrominance. It processes luminance signals and chrominance signals separately and is mainly used in analog color TV signals, video recording, encoding and decoding, etc.

[0172] This encoding method includes three components Y, U, V:

[0173] Y represents the brightness of the image, that is, the grayscale information, which indicates the brightness and darkness of the image. U represents the blue chromaticity, which refers to the difference between the blue signal and the brightness signal. V represents the red chromaticity, which refers to the difference between the red signal and the brightness signal.

[0174] Step B3: label the YUV format data subset using HEVC video encoding to obtain a data set with network training labels, thereby completing the production of the data set;

[0175] Among them, there are three layers of network training labels: the first layer is the network training label of whether 64x64CU can be divided into 32x32CU; the second layer is the network training label of whether 32x32CU can be divided into 16x16CU; the third layer is the network training label of whether 16x16CU can be divided into 8x8CU.

[0176] Specifically, the network training labels are the real labels of the data set under different quantization parameters; wherein the quantization parameters include four quantization levels, namely 22, 27, 32 and 37;

[0177] The normalization processing formula of the quantization parameter is as follows:

[0178]

[0179] Among them, q is the original quantization parameter QP value, is the normalized quantization parameter QP value.

[0180] Among them, q is the original quantization parameter QP value, is the normalized quantization parameter QP value. After conversion, the normalized value The range will become 0 to 1.

[0181] As a specific implementation, in step S01, the design loss function includes the following steps:

[0182] Step C1: According to the loss of each branch, calculate the sum of the losses of each branch Loss sum ;

[0183] Among them, the loss formula of the λth branch is as follows:

[0184]

[0185] Loss (λ) is the loss of the λth layer, χ is the normalization constant, y (λ) (G i,j ) represents the true value label of the λth layer, represents the predicted value of the λth layer, H(·) represents the cross entropy loss function, and H and W represent the height and width of the feature map respectively;

[0186] The sum of the losses of each branch Loss sum The calculation formula is as follows:

[0187]

[0188] Step C2: Calculate the cross entropy loss function for each layer based on the RD cost loss function The specific formula is as follows:

[0189]

[0190] is the normalized RD cost;

[0191] ζ represents the partitioning mode;

[0192] r ξ Indicates the RD cost of the nth CU of the current CTU under the partition mode ζ;

[0193] r min represents the minimum RD cost of CU in all possible partitioning modes;

[0194] Step C3: According to Loss sum and Get the overall loss function of the neural network, thereby completing the design of the loss function;

[0195]

[0196] Among them, L CPAC-I is the overall design function of the neural network.

[0197] The loss function is a function used to measure the difference or error between the model's predicted value and the true value. When training a machine learning model, the loss function is designed to enable the model to better fit the data.

[0198] The neural network combining attention and convolution constructed in this embodiment uses ACmix, traditional CNN or hole convolution combined with convolution and self-attention mechanism during feature extraction to capture images of different receptive field sizes. The rate distortion loss cost is introduced in combination with the cross entropy function to train the network. The network design and training methods are completely different. On the one hand, a fast HEVC video intra-frame CU block partitioning method based on convolution and attention fusion is designed to improve the original intra-frame coding strategy, and build a neural network called CPAC-I (CU partition network based on attention and convolution) that combines attention and convolution to replace the original CU partition decision algorithm in HEVC. Under the premise of ensuring that the video coding quality remains basically unchanged, the coding complexity of HEVC is reduced, the time cost required for video coding is reduced, the contradiction between coding quality and coding speed is alleviated, and the coding efficiency of HEVC is improved.

[0199] The technical solution adopted in this embodiment includes the following steps:

[0200] Step 1: Build CU partitioned CPAC-I network;

[0201] Drawing on the SKNet network architecture, CPAC-I is designed as a three-layer network parallel architecture, and a three-stage controlled decision method is adopted to achieve end-to-end classification of the network while terminating CU division in advance. According to the different functions implemented by the network, this embodiment divides the overall CPAC-I network into a feature extraction unit and a split decision unit. The feature extraction unit learns and extracts the key features in the image, and the split decision unit makes the division decision. Since the brightness component reflects the texture complexity of the CU, the input of CPAC-I includes the original brightness component and the classification label. Specifically, since the size of the normalized quantization parameter (Quantization Parameter, QP) will affect the final division result, before the split decision unit, QP values ​​of 22, 27, 32, and 37 are introduced as external feature variable inputs.

[0202] The QP value is the quantization parameter, which can also be called the quantization step. The lower the quantization step (QP), the finer the sampling in the quantization process. In theory, the image quality is higher and the size is relatively larger. On the contrary, the higher the quantization step, the lower the image quality, but the compression rate is improved and the file size is smaller.

[0203] Effect of various quantization steps

[0204] QP=22

[0205] Video quality: Very high, suitable for applications that require extremely high image quality (such as Blu-ray movies or high-quality streaming media).

[0206] File size: Larger, may result in lower compression.

[0207] Applicable scenarios: high-quality playback and post-production.

[0208] QP=27

[0209] Video quality: High, suitable for home theater, HD video needs, etc.

[0210] File size: Relatively small, while still retaining good quality.

[0211] Applicable scenarios: streaming services, high-quality video archiving.

[0212] QP=32

[0213] Video quality: Medium, suitable for general video viewing needs.

[0214] File Size: Compresses efficiently and usually does not significantly impact the viewing experience.

[0215] Applicable scenarios: online video platforms, daily video recording and sharing.

[0216] QP=37

[0217] Video quality: Low, may result in noticeable image distortion, especially in dynamic scenes.

[0218] File size: Significantly reduced, ideal for storage-constrained or low-bandwidth applications.

[0219] Constructing a CU-partitioned CPAC-I network includes the following steps:

[0220] (1) Build a feature extraction unit network:

[0221] First, take the brightness component L as the network input, Where C represents the number of channels. L is input into three branches respectively, corresponding to receptive fields of different sizes for feature learning, and finally extracted as image features under different receptive fields.

[0222] In the specific implementation, in branch 1 B1, L passes through a set of ACmix convolutions, a set of batch normalization BatchNormalization and ReLU activation functions, and then passes through two sets of 5×5 convolution kernels, a set of batch normalization and ReLU activation functions to obtain the intermediate feature map of B1 layer. In order to reduce the computational burden of the network, the 5×5 traditional convolution kernel is replaced by two 3×3 serial dilated convolutions.

[0223] In branch 2 B2, L passes through two sets of ACmix convolution kernels, a combination of batch normalization and ReLU activation function, and a set of 3×3 dilated convolution, followed by a combination of batch normalization and ReLU activation function, to obtain the output feature map of layer B2.

[0224] In branch three B3, L passes through three sets of ACmix convolution kernels, a combination of batch normalization and ReLU activation functions, and obtains the output feature map of layer B3

[0225] During the video encoding process, the quantization operation will cause the loss of video feature information, and the larger the QP value, the greater the loss of video encoding feature information, and more image details are lost, causing the CU to be more inclined to be divided into larger blocks. Therefore, the QP value is used as an external feature variable and input into the network.

[0226] Specifically, in the three branches of the network, before the feature maps are channel-fused, the intermediate features of each layer of the feature extraction unit first pass through a half-mask operation guided by the QP value, and the QP value is normalized to control its value range within the interval [0, 1].

[0227] Let the original QP value be q, and the normalized QP value be

[0228]

[0229] The input features are multiplied by the normalized QP value, which realizes the effect of the QP value on weakening the image information in the actual encoding operation, so that the network can more accurately determine the CU division under various QP values.

[0230] Next, in order to make the network adaptively adjust the size of the neuron receptive field according to the stimulus content, the features of the three branches are fused by element addition. The features of the three branches are fused by element addition to obtain the unfiltered fusion feature map G

[0231]

[0232] The collected features are then used to generate a channel statistics vector s through global average pooling. Specifically, the cth element in the channel statistics s is obtained by calculating the sum and average of the spatial dimensions in the feature map G in the range H×W:

[0233]

[0234] Next, s is fed into a fully connected layer to obtain a compact feature vector z to provide guidance for accurate and adaptive feature selection while reducing the dimensionality for improved efficiency.

[0235] z=δ(B(Ws)) (4)

[0236] where δ represents the ReLU function

[84] , B represents batch normalization

[83] , and the weights The value of d is controlled by a reduction rate r:

[0237]

[0238] In this embodiment, the value of r is 16, and L represents the minimum value of d.

[0239] Afterwards, the softmax operator is applied to the feature z, and three cross-channel soft attentions are guided according to the feature descriptor z to adaptively select information from different spatial scales, as shown in formula (6).

[0240]

[0241] in, u 1 ,u 2 ,u 3 Respectively represent U 1 ,U 2 ,U 3 The soft attention vector of represents the cth row of A, Respectively represent u 1 ,u 2 ,u 3 The cth element in .

[0242] Finally, the final feature map V of the three branches is obtained according to the attention weights of different kernels. (1) , V (2) and V (3) :

[0243]

[0244] The l-th layer feature map

[0245] Building a split decision unit

[0246] The three branch CTU feature maps extracted by the feature extraction unit are used as the input of the split decision unit. In the three branches of the split decision unit, the configuration of each split decision sub-network is related to the size of its corresponding predicted target CU. In order to extract the underlying features, each sub-network contains two convolutional layers. Since the length and width of the final CU division area are 2 n ,n∈(3,4,5,6), in order to make the receptive field of the convolution kernel consistent with the possible partitioned CU area in most cases, the width and height of all convolution layer kernels are set to 2 n ,n∈(1,2).

[0247] Specifically, the goal of the B1 layer is to determine whether the CTU is eventually divided into 32×32 blocks. The subnetwork contains two consecutive convolutional layers with the same convolution kernel size of 4×4. The goal of the B2 layer is to determine whether the CTU is eventually divided into 16×16 CU blocks. The subnetwork contains two consecutive convolutional layers with convolution kernel sizes of 4×4 and 2×2. The goal of the B3 layer is to determine whether the CTU is eventually divided into 8×8 CU blocks. The subnetwork contains two consecutive convolutional layers with a convolution kernel size of 2×2.

[0248] Next, in the three branches, although the convolution kernels of each layer are of different sizes, the two convolution layers corresponding to each layer are followed by two fully connected layers. The features after the convolution layer are sequentially transmitted to the two fully connected layers.

[0249] Finally, the classification labels of each level are output.

[0250] Specifically, judgment conditions are set between layers to control when the feature map flows into the split decision unit of each layer, so that the judgment of each layer is carried out in order from coarse to fine classification granularity. Specifically, starting from layer B1, if the judgment result is not to continue the division, the channel between the feature extraction unit of layer B2 and layer B3 and the split decision unit is closed, and the network operation ends at this time, and the judgment exits early. If the judgment result is to continue the division, the feature map of the feature extraction unit of layer B2 flows into the split decision unit, and the network continues to make division judgments until a certain layer is judged to be no longer divided or the final B3 layer division judgment ends, and the network terminates.

[0251] Step 2: Dataset Creation

[0252] The CU partition label dataset is created using the RAISE image dataset.

[0253] Firstly, the picture frames are randomly divided into a full I-frame video training set, a test set and a validation set with a ratio of 4:1:1. Furthermore, in order to better train the network to match the actual encoder, the images in the above three datasets are spliced ​​and converted into YUV format data subsets with resolutions of 768×512, 1536×1024, 1920×1080 and 2880×1920 respectively.

[0254] Next, the dataset labels were produced using the HEVC video encoder (HM-16.5) developed by the JVET Joint Video Experts Group.

[0255] In HM-16.5, the encoding configuration is set to All Intar (AI), and all data subsets are encoded with QP values ​​of 22, 27, 32, and 37. During the encoding process, all CTU divisions are counted, and the final CU division results are recorded in combination with the YUV current frame position coordinates. At the same time, the RDCost values ​​of each layer calculated during CU division are recorded as a basis for network training.

[0256] RD Cost refers to the rate-distortion cost (Rate-Distortion Cost) or RD cost, which is an indicator of coding efficiency in video coding. During the CU (coding unit) division process, the rate-distortion cost under different splitting decisions is calculated as an evaluation criterion to select the best CU division strategy.

[0257] The rate-distortion cost is composed of two factors:

[0258] Rate: refers to the bit rate required for encoding, usually used to indicate the compression efficiency of data, that is, the number of bits required after encoding. The lower the rate, the higher the compression efficiency.

[0259] Distortion: refers to the degree of distortion caused by encoding, indicating the difference or loss between the encoded image and the original image. The smaller the distortion, the smaller the quality loss.

[0260] Finally, the output results are bundled to produce network training labels. Specifically, from a formal point of view, the goal of the present invention is to predict Where H and W represent the height and width of the input image, respectively, and C represents the number of CU partition types. For each 64×64 CTU to be decomposed, in the original coding standard, the deepest path is to be divided into a CU of 8×8 size after three judgment divisions. Therefore, the proposed label division structure has 3 layers, which encode the CU division type with different granularities from coarse to fine. The first layer records whether the 64×64 CTU is divided into 32×32, the second layer records whether the 32×32 CU is divided into 16×16, and the third layer CU records whether the 16×16 CU is divided into 8×8. If the current CU division is judged to be divided, the position of the layer where the current CU is located is marked as 1; otherwise, it is marked as 0. In particular, if the current CU is judged to be not divided, the subsequent deep layer marks the point position as Nan.

[0261] Step 3: Training the CPAC-I network

[0262] The network of the present invention has the following characteristics: First, the network structure is a 3-branch multi-branch classification network, and our expectation is that the loss of the λth layer branch in the network is (λ) The parameters in the current layer are optimized, and the shared parameters are the sum of the losses of each branch. sum To optimize,

[0263]

[0264] Among them, y (λ) (G i ) represents the true value label of the λth layer, represents the predicted value of the λth layer, and H(·) represents the cross entropy loss function. It is worth noting that when the sub-region in the CTU is not divided into 32×32 sizes, the label y (2) (G i ) and y (3) (G i ) The corresponding area is empty, then Loss (2) and Loss (3) is empty; similarly, when the sub-region in the CTU is not divided into 16×16 size, the label y (3) (G i ) The corresponding area is empty, then Loss (3) Is empty.

[0265] On the other hand, in HEVC, different splitting modes usually lead to different RD cost losses, and the simple cross entropy function cannot represent this problem. Therefore, on the basis of calculating the cross entropy function at each layer, the RD cost loss function is introduced to calculate the final cross entropy loss function of each layer:

[0266]

[0267] Among them, r ξ represents the RD cost of the nth CU of the current CTU under the partition mode ζ, r min represents the minimum RD cost of the CU in all possible partitioning modes. In the above formula,

[0268]

[0269] can be viewed as the normalized RD cost, which is compared with the predicted value After multiplication, when a larger error prediction probability value or a larger RD cost is generated, a larger loss value will be generated, thus achieving the purpose of training the network. Combining (8) and (9), the overall loss function of the neural network is:

[0270]

[0271] This is used to train the network and save the network model.

[0272] Step 4: Embed the network model into the original HM16.5 encoder. Inside the encoder, when the CTU to be encoded needs to be divided, skip the original intra-frame CU division judgment, directly normalize the CTU image and input it into the trained model, complete the CU division process according to the judgment result of the network model, and perform subsequent changes, quantization, loop filtering, entropy coding and other encoding processes, and finally output the video encoding stream.

[0273] The self-attention mechanism can establish long-range dependencies and expand the receptive field of the image. Compared with CNN, its receptive field is larger and can obtain more contextual information. On the other hand, the self-attention mechanism is achieved by screening important information and filtering unimportant information, which makes its ability to capture effective information weaker than CNN. This is because the self-attention mechanism cannot take advantage of the scale and translation invariance of the image itself, as well as the locality of the image's features, that is, adjacent areas on the image have similar features, that is, the information of the same object is often concentrated locally. Based on this prior knowledge, learning can only be done through a large amount of data. Therefore, the self-attention mechanism can only effectively establish accurate global relationships based on big data, and when the amount of data is small, the effect is not as good as CNN. For a long time, convolution and self-attention have been two independent modules following different design paradigms.

[0274] Therefore, in practical applications, it is necessary to select a suitable partitioning method for encoding in combination with specific scenarios and data characteristics to obtain better encoding effects. At the same time, it is also necessary to weigh and optimize in combination with considerations such as hardware performance and algorithm complexity to meet practical application requirements. The CPAI-I network designed in this embodiment adopts the ACmix module to combine the convolution process with the self-attention process while avoiding two cumbersome projection operations, so as to better extract features while generating excessive computational complexity. At the same time, considering that in the video encoding process, the quantization operation will lead to the loss of video feature information. On the other hand, different splitting modes usually lead to different RD cost losses, and the network training using a simple cross entropy function cannot represent this problem. Therefore, the QP value is used as an external feature variable and input into the network, which is more suitable for model application. During network training, on the basis of calculating the cross entropy function at each layer, the loss function of the RD cost is introduced to calculate the final cross entropy loss function of each layer.

[0275] Embodiment 2:

[0276] This embodiment provides a HEVC video intra-frame coding method, the method comprising the following steps:

[0277] Step K1: using the HEVC intra-frame CU division method based on deep learning described in Example 1 to divide the HEVC frame;

[0278] Step K2: transform, quantize and entropy encode the divided HEVC frames to obtain encoding preprocessing data;

[0279] Step K3: Pack the encoding preprocessing data to obtain a video encoding stream, thereby completing HEVC video intra-frame encoding.

[0280] Embodiment 3:

[0281] like Figure 2 As shown, this embodiment provides a HEVC intra-frame CU division device based on deep learning, and the device includes:

[0282] An acquisition unit 10 is used to acquire video or image information to be divided into coding units;

[0283] A first processing unit 20, connected to the acquisition unit 10, is used to perform standardization processing on the video or image information to obtain a CTU image to be encoded;

[0284] The second processing unit 30 is connected to the first processing unit 20 and is used to input the CTU image into the CPAC-I model to obtain a CU partition decision;

[0285] The CPAC-I model is a neural network model based on the SKnet attention mechanism and convolution technology.

[0286] The third processing unit 40 is connected to the second processing unit 30, and is used to divide the HEVC frame according to the CU division decision to complete the deep learning-based intra-CU division of the HEVC frame.

[0287] As a specific implementation, the device further includes:

[0288] Building blocks for constructing the CPAC-I model;

[0289] The building block comprises:

[0290] A processing module, for preparing a data set based on historical video or image information, and for dividing the data set into a training set, a test set, and a validation set; and for designing a loss function;

[0291] a design module connected to the processing module, for training the CU partitioning CPAC-I network according to the training set based on the loss function; and testing the effect of the trained CU partitioning CPAC-I network according to the test set; and adjusting the parameters in the trained CU partitioning CPAC-I network according to the validation set, thereby obtaining a CPAC-I model;

[0292] Among them, the CU partition CPAC-I network is a pre-built neural network, and the CU partition CPAC-I network includes a feature extraction unit and a split decision unit. The feature extraction unit is used to learn and extract the key features of the image, and the split decision unit is used to make decisions on partitioning based on the key features of the image extracted through learning; the key features of the image are image features that affect the CTU image partition decision or the coding structure.

[0293] The device in this embodiment can execute the method in embodiment 1.

[0294] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A HEVC intra-frame CU division method based on deep learning, characterized in that: The method comprises the following steps: Step S1: Obtaining video or image information to be divided into coding units; Step S2: normalizing the video or image information to obtain a CTU image to be encoded; Step S3: inputting the CTU image into the CPAC-I model to obtain a CU partition decision; The CPAC-I model is a neural network model based on the SKnet attention mechanism and convolution technology. Step S4: Divide the HEVC frame according to the CU division decision to complete the deep learning-based intra-HEVC frame CU division.

2. The HEVC intra-frame CU division method based on deep learning according to claim 1, characterized in that: Before step S3, the method includes step S0; Step S0: constructing a CPAC-I model; The step S0 specifically includes the following steps: Step S01: creating a data set based on historical video or image information, and dividing the data set into a training set, a test set, and a validation set; and designing a loss function; Step S02: Based on the loss function, the CU partitioning CPAC-I network is trained according to the training set; and the effect of the trained CU partitioning CPAC-I network is tested according to the test set; and the parameters in the trained CU partitioning CPAC-I network are adjusted according to the validation set, so as to obtain a CPAC-I model; Among them, the CU partition CPAC-I network is a pre-built neural network, and the CU partition CPAC-I network includes a feature extraction unit and a split decision unit. The feature extraction unit is used to learn and extract the key features of the image, and the split decision unit is used to make decisions on partitioning based on the key features of the image extracted through learning; the key features of the image are image features that affect the CTU image partition decision or the coding structure.

3. The HEVC intra-frame CU division method based on deep learning according to claim 2, characterized in that: Before step S02, the method further includes step A: Step A: Construct CU partition CPAC-I network; The step A specifically comprises the following steps: Step A1: Take the brightness component L as the input of the neural network and input it into the three parallel branches of the neural network to obtain the first branch image feature quantity The second branch image feature The third branch image feature Step A2: Fuse the first branch image feature quantity, the second branch image feature quantity, and the third branch image feature quantity to obtain the total unfiltered fusion feature quantity G s ; in, represents the first branch image feature, represents the second branch image feature, represents the third branch image feature; Step A3: According to the total amount of unscreened fusion features G s , based on the attention weights of different kernels, the final feature map V of the three branches is obtained (1) , V (2) and V (3) ; Step A4: According to the final feature map V of the three branches (1) , V (2) and V (3) , the split decision is obtained to perform CU partitioning, thereby constructing a CU partitioning CPAC-I network.

4. The HEVC intra-frame CU division method based on deep learning according to claim 3, characterized in that: The step A3 specifically includes the following steps: Step A31: According to the total amount of unscreened fusion features G s , generate the characteristic quantity s of the cth element in the channel statistics vector s c ; Among them, G s (i,j) represents the total amount of unfiltered fusion features in the (i,j)th position; H and W represent the height and width of the feature map, respectively; Step A32: The feature quantity s c Send it to the fully connected layer to get a compact feature vector z; z=δ(B(Ws c )); Where δ represents the ReLU function, B() function represents batch normalization, and weight The formula for determining the value of d is as follows: Where r represents the reduction rate; d L represents the minimum value of d; Step A33: Apply the softmax operator to the feature vector z to obtain the soft attention vector in, u 1 ,u 2 ,u 3 Respectively represent U 1 ,U 2 ,U 3 The soft attention vector of represents the cth row of A, represents the cth row of B, represents the cth row of D, Respectively represent u 1 ,u 2 ,u 3 The cth element in ; Step A34: According to the soft attention vector Get the final feature map V of the three branches (1) , V (2) and V (3) ; The l-th layer feature map 5. The HEVC intra-frame CU division method based on deep learning according to claim 2, characterized in that: In step S01, the data set is prepared according to the historical video or image information, which specifically includes the following steps: Step B1: randomly shuffle the image frames in the historical video or image information, and then divide them into a training set, a test set, and a validation set in a ratio of 4:1:1; Step B2: splice and convert the images in the divided training set, test set and validation set into YUV format data subsets with different resolutions; Step B3: label the YUV format data subset using HEVC video encoding to obtain a data set with network training labels, thereby completing the production of the data set; Among them, the network training labels have three layers: the first layer is the network training label of whether 64x64CU can be divided into 32x32CU; the second layer is the network training label of whether 32x32CU can be divided into 16x16CU; the third layer is the network training label of whether 16x16CU can be divided into 8x8CU.

6. The HEVC intra-frame CU division method based on deep learning according to claim 5, characterized in that: In step B3, the network training labels are the real labels of the data set under different quantization parameters; wherein the quantization parameters include four quantization levels, namely 22, 27, 32 and 37; The normalization formula of the quantization parameter is as follows: Wherein, q is the original quantization parameter QP value, and q is the normalized quantization parameter QP value.

7. The HEVC intra-frame CU division method based on deep learning according to claim 2, characterized in that: In step S01, the design loss function includes the following steps: Step C1: According to the loss of each branch, calculate the sum of the losses of each branch Loss sum ; Among them, the loss formula of the λth branch is as follows: Loss (λ) is the loss of the λth layer, χ is the normalization constant, y (λ) (G i,j ) represents the true value label of the λth layer, represents the predicted value of the λth layer, H(·) represents the cross entropy loss function, and H and W represent the height and width of the feature map respectively; The sum of the losses of each branch Loss sum The calculation formula is as follows: Step C2: Calculate the cross entropy loss function for each layer based on the RD cost loss function The specific formula is as follows: is the normalized RD cost; ζ represents the partitioning mode; r ξ Indicates the RD cost of the nth CU of the current CTU under the partition mode ζ; r min represents the minimum RD cost of CU in all possible partitioning modes; Step C3: According to Loss sum and Get the overall loss function of the neural network, thereby completing the design of the loss function; Among them, L CPAC-I is the overall design function of the neural network.

8. A HEVC video intra-frame coding method, characterized in that: The method comprises the following steps: Step K1: using the HEVC intra-frame CU division method based on deep learning as described in any one of claims 1 to 7 to divide the HEVC frame; Step K2: transform, quantize and entropy encode the divided HEVC frames to obtain encoding preprocessing data; Step K3: Pack the encoding preprocessing data to obtain a video encoding stream, thereby completing HEVC video intra-frame encoding.

9. A HEVC intra-frame CU division device based on deep learning, characterized in that: The device comprises: An acquisition unit, used to acquire video or image information to be divided into coding units; A first processing unit, connected to the acquisition unit, configured to perform standardization processing on the video or image information to obtain a CTU image to be encoded; A second processing unit, connected to the first processing unit, configured to input the CTU image into a CPAC-I model to obtain a CU partition decision; The CPAC-I model is a neural network model based on the SKnet attention mechanism and convolution technology. The third processing unit is connected to the second processing unit, and is used to divide the HEVC frame according to the CU division decision to complete the deep learning-based intra-CU division of the HEVC frame.

10. The HEVC intra-frame CU division device based on deep learning according to claim 9, characterized in that: The device also includes: Building blocks for constructing the CPAC-I model; The building block comprises: A processing module, for preparing a data set based on historical video or image information, and for dividing the data set into a training set, a test set, and a validation set; and for designing a loss function; a design module connected to the processing module, for training the CU partitioning CPAC-I network according to the training set based on the loss function; and testing the effect of the trained CU partitioning CPAC-I network according to the test set; and adjusting the parameters in the trained CU partitioning CPAC-I network according to the validation set, thereby obtaining a CPAC-I model; Among them, the CU partition CPAC-I network is a pre-built neural network, and the CU partition CPAC-I network includes a feature extraction unit and a split decision unit. The feature extraction unit is used to learn and extract the key features of the image, and the split decision unit is used to make decisions on partitioning based on the key features of the image extracted through learning; the key features of the image are image features that affect the CTU image partition decision or the coding structure.

Images