Texture-based video coding method, equipment and medium
By extracting texture direction information and applying deep learning model prediction modes for H.264 encoding technology, the image quality and flexibility problems of H.264 encoding in complex textures and dynamically changing video scenes are solved, achieving more efficient and flexible video encoding.
Patent Information
- Application Number
- CN202410729393.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-06
- Publication Date
- 2025-02-18
Smart Images
Figure CN119484864A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video encoding, and in particular to a texture-based video encoding method, device and medium. Background Art
[0002] With the rapid development of digital media technology, the dissemination and consumption of video content on the Internet has reached an unprecedented scale. High-definition video has become increasingly popular because it provides a better visual experience, which has directly driven the demand for efficient video compression technology. As a widely used video coding standard, H.264 / AVC has become the mainstream choice in the industry with its good compression efficiency and compatibility. However, in the existing technology, H.264 encoding still faces many challenges, especially when processing video scenes with complex textures or dynamic changes.
[0003] First, although H.264 encoding can provide a high compression ratio, it often consumes more bit rate to maintain image quality when processing video data with complex textures. Complex textures contain a lot of detailed information, such as fast-moving scenes or environments with complex light changes, which are prone to block effects or blurring during compression, especially when the bit rate is limited.
[0004] Secondly, the parameter configuration of H.264 encoding is usually relatively fixed and lacks sufficient flexibility to adapt to different types of video content. This static encoding strategy is difficult to meet the special needs of various scenarios while ensuring encoding efficiency. For example, static scenes and dynamic scenes have very different requirements for encoding strategies. The former may be more suitable for using lower encoding complexity and bit rate, while the latter requires high bit rate and complex encoding technology to capture detailed dynamic changes.
[0005] Furthermore, real-time video encoding is particularly important in network video transmission, such as video conferencing and live broadcasting. These application scenarios require the encoder to not only process video frames quickly, but also dynamically adjust the encoding quality and flow according to network conditions to adapt to network characteristics such as bandwidth fluctuations and delays. However, the existing H.264 encoding strategy is still limited in dynamically adapting to network conditions, which may affect the smoothness of video transmission and viewing experience.
[0006] Finally, although the existing coding technology has achieved certain achievements in compression efficiency, in the large-scale real-time processing and transmission of high-quality videos, how to further improve the compression efficiency while ensuring the video quality is still an important direction for technological development. Therefore, in view of the shortcomings of the existing H.264 coding technology, the development of new video coding strategies, especially methods that can fully utilize the characteristics of video content for intelligent coding, has become an urgent problem that the industry needs to solve. Summary of the invention
[0007] The present invention aims to solve at least one of the technical problems existing in the prior art.
[0008] To this end, a first aspect of the present invention provides a texture-based video encoding method.
[0009] A second aspect of the present invention provides a computer device.
[0010] A third aspect of the present invention provides a computer-readable storage medium.
[0011] The present invention provides a texture-based video encoding method, which is applied to an H.264 encoding environment. The method comprises:
[0012] Gradient calculation is performed on all pixels of the intra-coded frame to extract the texture direction information of the pixels;
[0013] Using a trained deep learning model, predicting an optimal intra-frame prediction mode for each pixel block based on the calculated gradient information; wherein the deep learning model is trained based on different types of video data, and the types of video data include motion video, static scene, and low-light video;
[0014] According to the best intra-frame prediction mode output by the model, the coding units are classified into smooth units and rough units; wherein the smooth units represent units with consistent textures, and the rough units represent units with complex textures;
[0015] Different coding strategies are adopted for the classified coding units, where the smooth unit is encoded at a lower bit rate to reduce the amount of data, and the rough unit is encoded at a higher bit rate to preserve the details;
[0016] Monitor and analyze coding efficiency and quality in real time, and dynamically adjust prediction mode and coding unit division strategy.
[0017] The texture-based video encoding method according to the above technical solution of the present invention may also have the following additional technical features:
[0018] In the above technical solution, the step of extracting pixel texture direction information according to the gradient calculation result includes:
[0019] Calculate the horizontal and vertical gradients for each pixel;
[0020] The texture direction of each pixel is determined using the calculated horizontal and vertical gradients, where the texture direction is calculated as:
[0021] θ=arctan2(G y , G x )
[0022] Where θ represents the texture direction angle; G y Represents the vertical gradient, that is, the brightness change of the image in the horizontal direction; G x Represents the horizontal gradient, which is the brightness change of the image in the vertical direction.
[0023] In the above technical solution, the deep learning model is a convolutional neural network, and the convolutional neural network includes:
[0024] Convolutional layers, used to extract features from video frames;
[0025] Pooling layer, used to reduce feature dimensions and increase the generalization ability of the model;
[0026] The fully connected layer is used to convert the extracted features into the final prediction output, which is the best intra-frame prediction mode for each pixel block.
[0027] In the above technical solution, the method for training the deep learning model includes:
[0028] Collect and annotate video datasets for training, including the best intra-frame prediction mode for each pixel block;
[0029] The convolutional neural network is trained by supervised learning using annotated data sets. During the training process, the network parameters are continuously updated through the back-propagation algorithm to minimize the prediction error.
[0030] Perform model validation to test the model's predictive accuracy using data that was not used in training, and adjust and optimize the model as needed.
[0031] In the above technical solution, the deep learning model uses the cross entropy loss function as the loss function, and uses the cross entropy loss function to measure the difference between the probability distribution of the model output and the probability distribution of the true label. The optimization goal is to minimize this difference; the cross entropy loss function is:
[0032]
[0033] Among them, y i is the true label vector of the i-th prediction mode; is the probability vector predicted by the model for the i-th prediction mode; C is the total number of prediction mode types; the types of prediction modes include plane, vertical, and horizontal.
[0034] In the above technical solution, the attribute classification of the coding unit according to the best intra-frame prediction mode output by the model includes:
[0035] Analyze the prediction patterns of each pixel block output by the deep learning model;
[0036] The prediction mode distribution in each coding unit is counted. When the proportion of pixel blocks with the same prediction mode in the coding unit exceeds a set threshold, the coding unit is classified as a smooth unit; otherwise, the coding unit is classified as a rough unit.
[0037] In the above technical solution, the Sobel operator is used to calculate the gradient information. The Sobel operator calculates the horizontal and vertical gradients of the image respectively through convolution operation, thereby extracting the edge information of the image.
[0038] In the above technical solution, after the coding units are attribute-classified according to the best intra-frame prediction mode output by the model, the method further includes:
[0039] Identify continuous and adjacent smooth cells;
[0040] Adjacent smoothing units are merged, and the bit rate of the coding strategy adopted for the merged smoothing unit is lower than the bit rate of the coding strategy adopted for the smoothing unit before the merger.
[0041] The present invention also provides a computer device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded and executed by the processor, a texture-based video encoding method as described in any one of the above technical solutions is implemented.
[0042] The present invention also provides a computer-readable storage medium storing a program, which, when loaded by a processor, implements the texture-based video encoding method as described in any one of the above technical solutions.
[0043] In summary, due to the adoption of the above technical features, the beneficial effects of the present invention are:
[0044] The present invention significantly improves the efficiency and quality of H.264 encoding by introducing deep learning technology and cross entropy loss function. The method enables the encoder to more accurately predict the intra-frame coding mode, thereby reducing unnecessary calculations and improving the encoding speed. At the same time, due to the improved accuracy of the prediction, the quality of the encoded video is also guaranteed. In addition, the method of the present invention has good adaptability and can process different types of video content, including videos with intense actions and static scene videos.
[0045] Additional aspects and advantages of the present invention will become apparent from the following description or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0047] Figure 1The figure is a flowchart of a texture-based video encoding method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0048] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0049] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.
[0050] Refer to the following Figure 1 The texture-based video encoding method provided according to some embodiments of the present invention is described.
[0051] Some embodiments of the present application provide a texture-based video encoding method.
[0052] like Figure 1 As shown, the first embodiment of the present invention proposes a texture-based video encoding method, which is applied to the H.264 encoding environment. The method specifically includes steps S1-S5.
[0053] S1. Perform gradient calculation on all pixels of an intra-coded frame (I frame) to extract texture direction information of the pixels.
[0054] Gradient calculation reveals the intensity and direction of pixel value changes, which is the key to understanding image texture. Specifically, in step S1, the step of extracting pixel texture direction information according to the gradient calculation result includes:
[0055] Calculate the horizontal and vertical gradients for each pixel;
[0056] The texture direction of each pixel is determined using the calculated horizontal and vertical gradients. Specifically, the texture direction angle is determined by the inverse tangent function. The texture direction is calculated as follows:
[0057] θ=arctan2(G y , G x )
[0058] Where θ represents the texture direction angle; G y Represents the vertical gradient, that is, the brightness change of the image in the horizontal direction; G x Represents the horizontal gradient, which is the brightness change of the image in the vertical direction.
[0059] In some embodiments, a Sobel operator may be used to calculate gradient information. The Sobel operator calculates horizontal and vertical gradients of an image respectively through a convolution operation, thereby extracting edge information of the image.
[0060] S2. Use a trained deep learning model to predict the optimal intra-frame prediction mode for each pixel block based on the calculated gradient information; wherein the deep learning model is trained based on different types of video data, and the types of video data include motion video, static scene and low-light video.
[0061] Specifically, the deep learning model is a convolutional neural network, which includes: an input layer, a convolutional layer, a pooling layer and a fully connected layer. In the input layer, the model receives a single or multiple consecutive intra-frame coded frames as input, and the input data includes the original pixel value and its gradient information. The convolutional layer is used to extract features in the video frame. Specifically, multiple convolutional layers are used to extract features in the video frame, and each convolutional layer is usually followed by an activation function, such as ReLU, to enhance nonlinear learning capabilities. The pooling layer is used to reduce the feature dimension and increase the generalization ability of the model. Maximum pooling can be used to keep the important parts of the features. The fully connected layer is used to convert the extracted features into the final prediction output, that is, the optimal intra-frame prediction mode for each pixel block.
[0062] In some embodiments, a method of training a deep learning model includes:
[0063] Collect and annotate video datasets for training. The annotation information includes the best intra-frame prediction mode for each pixel block. The video dataset should include different types of video data, such as motion video, static scene, low-light video, etc. When annotating, the optimal prediction mode for each pixel block in the existing coding environment should be manually or automatically marked according to the H.264 coding standard, such as planar, vertical, horizontal, etc.
[0064] The convolutional neural network is trained by supervised learning using an annotated data set. During the training process, the network parameters are continuously updated through the back propagation algorithm to minimize the prediction error. Specifically, the data set is split using the cross-validation method for model training. During the training process, a loss function (such as cross entropy loss) is used to evaluate the difference between the predicted mode output by the model and the actual optimal mode, and the network parameters are continuously adjusted through an optimization algorithm (such as Adam or SGD) to minimize the loss.
[0065] Perform model validation, test the model's prediction accuracy using data that was not used in training, and adjust and optimize the model as needed. Find the best network structure and training configuration by adjusting hyperparameters such as learning rate, batch size, number and size of convolutional layers, etc.
[0066] In a specific embodiment, the deep learning model uses a cross entropy loss function as a loss function, and uses the cross entropy loss function to measure the difference between the probability distribution of the model output and the probability distribution of the true label, and the optimization goal is to minimize this difference; the cross entropy loss function is:
[0067]
[0068] Among them, y i is the true label vector of the i-th prediction mode; is the probability vector predicted by the model for the i-th prediction mode; C is the total number of prediction mode types; the types of prediction modes include plane, vertical, and horizontal.
[0069] S3. Classify the coding units into smooth units and rough units according to the best intra-frame prediction mode output by the model; wherein the smooth units represent units with consistent textures, and the rough units represent units with complex textures.
[0070] Among them, smooth cells usually show high pattern consistency (such as most predictions are planar patterns), while coarse cells have complex textures and scattered pattern predictions. It can be understood that the pixel blocks in smooth cells have high pattern similarity, while coarse cells show high texture complexity.
[0071] In some embodiments, the attribute classification of the coding unit according to the best intra-frame prediction mode output by the model in step S3 includes:
[0072] Analyze the prediction patterns of each pixel block output by the deep learning model;
[0073] The prediction mode distribution in each coding unit is counted. When the proportion of pixel blocks with the same prediction mode in the coding unit to the total number of pixel blocks exceeds a set threshold, the coding unit is classified as a smooth unit; otherwise, the coding unit is classified as a rough unit. The set threshold of the proportion of pixel blocks with the same prediction mode in the coding unit to the total number of pixel blocks can be adjusted according to the quality and efficiency requirements, and a suitable value is selected for determination. In a specific embodiment, the initial value of the set threshold is set to 80%.
[0074] In some embodiments, after classifying the attributes of the coding units according to the best intra-frame prediction mode output by the model, the method further includes:
[0075] Identify continuous and adjacent smooth cells;
[0076] The adjacent smoothing units are merged, and the bit rate of the coding strategy adopted for the merged smoothing unit is lower than the bit rate of the coding strategy adopted by the smoothing unit before the merger. The above method can reduce the overall complexity of the coding. The merged large unit is encoded at a lower bit rate than the original individual small unit, further optimizing the data transmission and storage efficiency.
[0077] S4, adopting different encoding strategies for the classified coding units, wherein the smooth unit is encoded at a lower bit rate to reduce the amount of data, and the rough unit is encoded at a higher bit rate to preserve details;
[0078] Specifically, a lower bit rate is used for encoding smooth cells, because these areas have less visual changes, are easier to compress, and are less likely to be perceived by the observer. A higher bit rate is used for rough cells to ensure that textures and details are preserved, especially in more visually significant areas.
[0079] S5. Monitor and analyze coding efficiency and quality in real time, and dynamically adjust prediction mode and coding unit division strategy.
[0080] Some other embodiments of the present invention provide a computer device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is loaded and executed by the processor, the texture-based video encoding method as described in the above embodiments is implemented.
[0081] Some other embodiments of the present invention provide a computer-readable storage medium storing a program, which, when loaded by a processor, implements the texture-based video encoding method as described in the above embodiments.
[0082] In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.
[0083] Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A texture-based video coding method, applied to H.264 coding environment, characterized in that: The method comprises: Gradient calculation is performed on all pixels of the intra-coded frame to extract the texture direction information of the pixels; Using a trained deep learning model, predicting an optimal intra-frame prediction mode for each pixel block based on the calculated gradient information; wherein the deep learning model is trained based on different types of video data, and the types of video data include motion video, static scene, and low-light video; According to the best intra-frame prediction mode output by the model, the coding units are classified into smooth units and rough units; wherein the smooth units represent units with consistent textures, and the rough units represent units with complex textures; Different coding strategies are adopted for the classified coding units, where the smooth unit is encoded at a lower bit rate to reduce the amount of data, and the rough unit is encoded at a higher bit rate to preserve the details; Monitor and analyze coding efficiency and quality in real time, and dynamically adjust prediction mode and coding unit division strategy.
2. The texture-based video encoding method according to claim 1, characterized in that: The steps of extracting pixel texture direction information according to the gradient calculation result include: Calculate the horizontal and vertical gradients for each pixel; The texture direction of each pixel is determined using the calculated horizontal and vertical gradients, where the texture direction is calculated as: θ=arctan2(G y ,G x ) Where θ represents the texture direction angle; G y Represents the vertical gradient, that is, the brightness change of the image in the horizontal direction; G x Represents the horizontal gradient, which is the brightness change of the image in the vertical direction.
3. The texture-based video encoding method according to claim 1, characterized in that: The deep learning model is a convolutional neural network, which includes: Convolutional layers, used to extract features from video frames; Pooling layer, used to reduce feature dimensions and increase the generalization ability of the model; The fully connected layer is used to convert the extracted features into the final prediction output, which is the best intra-frame prediction mode for each pixel block.
4. The texture-based video encoding method according to claim 3, characterized in that: Methods for training deep learning models include: Collect and annotate video datasets for training, including the best intra-frame prediction mode for each pixel block; The convolutional neural network is trained by supervised learning using annotated data sets. During the training process, the network parameters are continuously updated through the back-propagation algorithm to minimize the prediction error. Perform model validation to test the model's predictive accuracy using data that was not used in training, and adjust and optimize the model as needed.
5. The texture-based video encoding method according to claim 4, characterized in that: The deep learning model uses the cross entropy loss function as the loss function, and uses the cross entropy loss function to measure the difference between the probability distribution of the model output and the probability distribution of the true label. The optimization goal is to minimize this difference; the cross entropy loss function is: Among them, y i is the true label vector of the i-th prediction mode; is the probability vector predicted by the model for the i-th prediction mode; C is the total number of prediction mode types; the types of prediction modes include plane, vertical, and horizontal.
6. The texture-based video encoding method according to claim 1, characterized in that: The classifying the attributes of the coding units according to the best intra-frame prediction mode output by the model includes: Analyze the prediction patterns of each pixel block output by the deep learning model; The prediction mode distribution in each coding unit is counted. When the proportion of pixel blocks with the same prediction mode in the coding unit exceeds a set threshold, the coding unit is classified as a smooth unit; otherwise, the coding unit is classified as a rough unit.
7. The texture-based video encoding method according to claim 1, characterized in that: The Sobel operator is used to calculate the gradient information. The Sobel operator calculates the horizontal and vertical gradients of the image through convolution operations, thereby extracting the edge information of the image.
8. The texture-based video encoding method according to claim 1, characterized in that: After classifying the attributes of the coding units according to the best intra prediction mode output by the model, it also includes: Identify continuous and adjacent smooth cells; Adjacent smoothing units are merged, and the bit rate of the coding strategy adopted for the merged smoothing unit is lower than the bit rate of the coding strategy adopted for the smoothing unit before the merger.
9. A computer device, characterized in that: The invention comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded and executed by the processor, the texture-based video encoding method according to any one of claims 1 to 8 is implemented.
10. A computer-readable storage medium, characterized in that: A program is stored, and when the program is loaded by a processor, the texture-based video encoding method according to any one of claims 1 to 8 is implemented.
Citation Information
Cited By
Artificial intelligence training method and system based on cloud computing
CN115758262A