Visually controllable coding end deep learning video compression control method
By pre-training the deep learning codec on the encoding side and introducing coding mode and frame structure prediction strategies, it optimizes for downstream video analysis tasks, which solves the problem that existing video compression methods cannot effectively reduce bit rate redundancy, and achieves support and performance improvements for a variety of downstream tasks.
Patent Information
- Application Number
- CN202411622248.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-14
- Publication Date
- 2025-03-14
AI Technical Summary
The existing video compression methods are mainly aimed at human eye vision optimization, and cannot effectively reduce the redundancy of code rate in downstream video analysis tasks. The existing video encoding methods for machine vision need to design codecs for each task, which increases the complexity and deployment difficulty.
The deep learning codec for human eye vision optimization is pre-trained on the encoding side, and a coding mode prediction and coding frame structure prediction strategy are added to train and optimize for downstream tasks to form an integrated video codec.
It supports a variety of downstream video analysis tasks without changing the original codec weight and decoding process, which significantly reduces the bit rate redundancy and improves the bit rate-accuracy performance.
Smart Images
Figure CN119629360A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and in particular to a visually controllable encoding-end deep learning video compression control method. Background Art
[0002] Video compression coding technology is widely used in various information systems. In recent years, with the rapid development of deep learning technology, video compression coding technology based on deep learning has made great progress. Video encoders based on deep learning can effectively eliminate redundant semantic information of images. Compared with traditional codecs, they have achieved significant advantages in image quality and bit rate in specific scenarios. Recently, video analysis technologies, such as video target detection and video recognition, have been widely used in various fields of society, such as smart cities, autonomous driving, traffic monitoring planning, etc. After being collected, a large number of videos are compressed and encoded into binary code streams and transmitted to the cloud. Then the cloud is transmitted to the receiving end decoder to obtain decoded videos for downstream video analysis tasks. According to a Cisco report, more and more videos are directly applied to downstream analysis tasks after decoding, rather than directly used for human viewing, accounting for more than 50%. However, most of the existing video compression methods, including traditional video compression and deep learning-based video compression methods, are mainly optimized for human visual viewing quality. However, there are differences between downstream machine vision and human vision. For example, human vision focuses more on the high-definition reconstruction quality of video details, while tasks such as video object detection focus more on the target area and pay little attention to the reconstruction of details in the background area. Therefore, for downstream video analysis tasks, traditional compression encoding methods are suboptimal and bring about large encoding bit rate redundancy.
[0003] To meet this challenge, researchers defined it as the "Video Coding for Machine" task (VCM), which aims to enable the encoder to achieve high-quality decoded video while also being able to optimize downstream machine vision tasks such as video analysis and reduce bitrate redundancy. However, in order to optimize different tasks, existing VCM methods often require the design of a set of encoders and decoders for each task, and the joint optimization of codecs for different tasks. Although this method can support a variety of downstream video analysis tasks, such as video target detection and multi-target tracking, it also increases the complexity of the encoding and decoding process and the difficulty of deploying codecs, including deploying multiple codecs on the encoding and decoding ends, and requiring collaboration between different codecs, which makes actual deployment and application difficult. Summary of the invention
[0004] In order to solve the above technical problems, the present invention provides a visually controllable encoding end deep learning video compression control method, comprising:
[0005] Pre-train a deep learning codec optimized for human vision and freeze the model parameters to form the encoding end;
[0006] Adding a coding mode prediction strategy to the coding end, performing training optimization for downstream tasks, and obtaining an optimized coding mode prediction model;
[0007] The coding mode prediction model is trained, and after the training is completed, the coding frame structure prediction model is introduced, and the training is continued for downstream tasks to obtain a video codec that integrates coding mode prediction and coding frame structure prediction, and video decompression is implemented based on the video codec.
[0008] Preferably, the process of pre-training a deep learning codec optimized for human vision includes:
[0009] The residual video codec is pre-trained separately, and the conditional video codec is pre-trained using the pre-trained model weights to obtain a deep learning codec optimized for human vision.
[0010] Preferably, a coding mode prediction strategy is added to the coding end, and training optimization is performed for downstream tasks. The process of obtaining an optimized coding mode prediction model includes:
[0011] A coding mode prediction strategy is added to the encoding end, and multi-target tracking and video target detection tasks are selected for training respectively. Reasoning verification is performed on three video analysis tasks of multi-target tracking, video target detection and video action recognition to obtain an optimized coding mode prediction model.
[0012] Preferably, the loss function expression of the coding mode prediction model is:
[0013] L m =R+λ m L t
[0014] Among them, R is the bit rate required to encode the current frame, L t is the loss function size of the downstream task corresponding to the decoded frame, λ m A hyperparameter to balance the two.
[0015] Preferably, the process of performing model training on the coding mode prediction model includes:
[0016] The coding mode prediction model uses the distribution parameter prior information of the coding feature estimation of the video encoder entropy model as input to the residual convolutional neural network;
[0017] The residual convolutional neural network determines the importance of each position feature to the downstream machine vision task according to the distribution parameters of each position feature;
[0018] For the feature map to be encoded, the residual convolutional neural network predicts a coding mode of "0" or "1" for each element in the feature map, and uses the Gumbel-Softmax sampling strategy to perform sampling according to the probability output by the residual convolutional neural network, and differentially selects a mode with a larger probability value as the coding mode to form a coding mode binary mask with the same shape as the feature map.
[0019] Preferably, for the feature map to be encoded, the process of the residual convolutional neural network predicting the encoding mode of "0" or "1" for each element in the feature map includes:
[0020] For each element, if the prediction is "1", it means that the element is more important for the downstream visual task or it is difficult for the decoder to accurately predict the element based on the prior information of the entropy model, and the position needs to be encoded and transmitted;
[0021] On the contrary, if the prediction is "0", it means that the downstream task is not sensitive to the element, or the decoding end can accurately predict the element based on the prior information of the entropy model, and there is no need to encode and transmit the position.
[0022] Preferably, the loss function expression of the coding frame structure prediction model is:
[0023]
[0024] in, The average bit rate required to encode the predicted frames in the current picture group, is the average loss function size of the downstream task corresponding to the decoded prediction frame of the picture group, λ g A hyperparameter to balance the two.
[0025] Preferably, the coding mode prediction model is trained, and after the training is completed, the coding frame structure prediction model is introduced, and the process of continuing the training for downstream tasks includes:
[0026] The frame structure prediction training at the encoding end is carried out in units of picture groups. Multi-target tracking and video target detection tasks are also selected for training respectively. Reasoning verification is carried out on three video analysis tasks, namely multi-target tracking, video target detection and video action recognition, to obtain a video codec that integrates coding mode prediction and coding frame structure prediction.
[0027] Preferably, the process of performing frame structure prediction training at the encoding end in units of picture groups includes:
[0028] After obtaining the to-be-encoded picture group, firstly using the optical flow estimation network to perform inter-frame optical flow estimation based on the encoding frame structure prediction model to obtain the inter-frame optical flow;
[0029] Subsequently, based on the coding frame structure prediction model, the YOLO detector is used to process the picture group frame by frame to generate a target area mask generated by the overlap of the detection frame, wherein the value of the area inside the frame is set to "1" and the value of the area outside the frame is set to "0". The target area mask of each predicted frame is multiplied by the inter-frame optical flow corresponding to it to obtain the optical flow information after the mask, and the optical flow information is input into the coding frame structure prediction model again, and the optical flow information and the inter-frame residual information are used as input, and finally a binary tensor with a length equal to the number of predicted frames is output, and the prediction value at each position is "1" to represent that the coding type of the position is predicted to be a P frame, and the prediction value is "0" to represent that the coding type of the position is predicted to be a P frame. m frame;
[0030] The coding frame structure prediction model also uses the Gumbel-Softmax sampling strategy to implement gradient calculation and back propagation during training.
[0031] Compared with the prior art, the present invention has the following advantages and technical effects:
[0032] Based on the two existing mainstream video codecs, the "residual deep learning codec" and the "conditional deep learning codec", this paper designs a new visually controllable codec framework. Through coding mode prediction, coding frame structure prediction and other technologies at the encoding end, the encoder supports multiple downstream tasks without changing the weights and decoding process of the original codec, reducing the complexity of the coding and decoding process and deployment applications. Compared with existing methods, significant performance improvements have been achieved in downstream video analysis tasks.
[0033] The present invention is based on multiple existing video encoding and decoding models, and has lower training cost and stronger versatility. The present invention can support downstream video analysis tasks and human viewing by applying only a single decoder and without changing the decoding strategy. There is no need to follow the strategy of designing different decoders for different characters in the previous technology, and it has lower deployment and application difficulty. In addition, in video analysis tasks such as multi-target tracking, the present invention can achieve stronger bit rate-accuracy performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0035] Figure 1 A schematic diagram of a coding mode prediction model according to an embodiment of the present invention;
[0036] Figure 2 Schematic diagram of prediction under different picture group structures according to an embodiment of the present invention;
[0037] Figure 3 Schematic diagram of a frame structure prediction model at the encoding end according to an embodiment of the present invention. DETAILED DESCRIPTION
[0038] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0039] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0040] like Figure 1-3 As shown, this embodiment provides a visually controllable encoder-side deep learning video compression control method, including:
[0041] Pre-train a deep learning codec optimized for human vision and freeze the model parameters to form the encoding end;
[0042] Add a coding mode prediction strategy to the encoding end, perform training optimization for downstream tasks, and obtain an optimized coding mode prediction model;
[0043] The coding mode prediction model is trained. After the training is completed, the coding frame structure prediction model is introduced and the training is continued for downstream tasks to obtain a video codec that integrates coding mode prediction and coding frame structure prediction, and video decompression is implemented based on the video codec.
[0044] Furthermore, the process of pre-training a deep learning codec optimized for human vision includes:
[0045] The residual video codec is pre-trained separately, and the conditional video codec is pre-trained using the pre-trained model weights to obtain a deep learning codec optimized for human vision.
[0046] This embodiment proposes to control the encoder and encoding process at the encoding end based on a pre-trained deep learning codec, and use a single pre-trained decoder to support downstream tasks of human vision and machine vision.
[0047] Furthermore, a coding mode prediction strategy is added to the coding end, and training optimization is performed for downstream tasks. The process of obtaining an optimized coding mode prediction model includes:
[0048] A coding mode prediction strategy is added to the encoding end, and multi-target tracking and video target detection tasks are selected for training respectively. Reasoning verification is performed on three video analysis tasks: multi-target tracking, video target detection, and video action recognition, to obtain an optimized coding mode prediction model.
[0049] Furthermore, the loss function expression of the coding mode prediction model is:
[0050] L m =R+λ m L t
[0051] Among them, R is the bit rate required to encode the current frame, L t is the loss function size of the downstream task corresponding to the decoded frame, λ m A hyperparameter to balance the two.
[0052] Furthermore, the process of training the coding mode prediction model includes:
[0053] like Figure 1 As shown in the figure, in the encoding process, a coding mode prediction model for downstream tasks is introduced. The coding mode prediction model uses the distribution parameter prior information of the video encoder entropy model to estimate the coding features as input and passes it into the residual convolutional neural network;
[0054] The residual convolutional neural network determines the importance of each position feature to the downstream machine vision task based on its distribution parameters;
[0055] For the feature map to be encoded, the residual convolutional neural network predicts a coding mode of "0" or "1" for each element in the feature map, and uses the Gumbel-Softmax sampling strategy to sample according to the probability of the residual convolutional neural network output. The mode with a larger probability value is selected by differentiation as the coding mode to form a coding mode binary mask with the same shape as the feature map.
[0056] To further optimize the solution, in order to obtain the binary mask of the coding mode, the prediction network needs to predict the probability of applying two coding modes for each element, and select the mode with a larger probability value as the coding mode. However, in the neural network training process, the operation of "selecting the mode with a larger probability" is not differentiable, which leads to the problem that the gradient cannot be calculated and back-propagated normally. In order to solve this problem, this embodiment introduces the Gumbel-Softmax sampling strategy, which samples according to the probability of the network output, making the process differentiable, thereby realizing the calculation and back-propagation of the gradient.
[0057] Furthermore, for the feature map to be encoded, the process of the residual convolutional neural network predicting the encoding mode of "0" or "1" for each element in the feature map includes:
[0058] For each element, if the prediction is "1", it means that the element is more important for the downstream visual task or it is difficult for the decoder to accurately predict the element based on the prior information of the entropy model, and the position needs to be encoded and transmitted;
[0059] On the contrary, if the prediction is "0", it means that the downstream task is not sensitive to the element, or the decoding end can accurately predict the element based on the prior information of the entropy model, and there is no need to encode and transmit the position.
[0060] Furthermore, if Figure 2 As shown in the figure, from top to bottom are the picture group structure used by the existing deep learning encoder, the manually assigned picture group structure, and the dynamic prediction picture group structure; when encoding the video picture group (Group of Pictures, GoP), the existing deep learning video encoder divides the video frames into two types: I frames (reference frames) and P frames (prediction frames). This embodiment uses coding mode prediction to divide the original P frames into P frames (not using coding mode prediction) and P m In order to further optimize the rate-accuracy performance of the decoded video in the downstream tasks, this embodiment introduces a frame structure prediction strategy at the encoding end. According to this strategy, the frame type to be encoded is predicted, and the following is obtained: Figure 2 The dynamic frame structure is shown below.
[0061] Furthermore, the loss function expression of the coding frame structure prediction model is:
[0062]
[0063] in, The average bit rate required to encode the predicted frames in the current picture group, is the average loss function size of the downstream task corresponding to the decoded prediction frame of the picture group, and λg is a hyperparameter to balance the two.
[0064] Furthermore, the coding mode prediction model is trained, and after the training is completed, the coding frame structure prediction model is introduced. The process of continuing the training for downstream tasks includes:
[0065] The frame structure prediction training at the encoding end is carried out in units of picture groups. Multi-target tracking and video target detection tasks are also selected for training respectively. Reasoning verification is carried out on three video analysis tasks, namely multi-target tracking, video target detection and video action recognition, to obtain a video codec that integrates coding mode prediction and coding frame structure prediction.
[0066] Furthermore, the process of performing frame structure prediction training at the encoding end in units of picture groups includes:
[0067] like Figure 3 As shown in FIG. 1 , the coding frame structure prediction model mainly consists of two parts, namely the pre-analysis part and the structure prediction part. After obtaining the picture group to be coded, the optical flow estimation network is first used to estimate the inter-frame optical flow based on the coding frame structure prediction model to obtain the inter-frame optical flow;
[0068] Subsequently, the YOLO detector is used to process the picture group frame by frame based on the coded frame structure prediction model to generate a target area mask generated by the overlap of the detection frame, where the value of the area inside the frame is set to "1" and the value of the area outside the frame is set to "0". The target area mask of each predicted frame is multiplied by its corresponding inter-frame optical flow to obtain the optical flow information after the mask, and the optical flow information is input into the coded frame structure prediction model again. The optical flow information and the inter-frame residual information are used as input, and finally a binary tensor with the same length as the number of predicted frames is output. The predicted value at each position is "1" for the predicted encoding type of the position to be P frame, and the predicted value is "0" for the predicted encoding type of the position to be P m frame;
[0069] The coding frame structure prediction model also uses the Gumbel-Softmax sampling strategy to realize gradient calculation and back propagation during training.
[0070] The coding frame structure prediction model includes residual convolution blocks, pooling layers, and convolution layers.
[0071] This embodiment reduces the bitstream redundancy for downstream machine vision tasks by introducing a coding mode prediction strategy; then introduces a coding end frame structure prediction strategy to control the coding method, thereby improving the bitrate-accuracy performance of the decoded video in downstream tasks.
[0072] The visually controllable deep learning compression coding method proposed in the present invention introduces coding mode prediction and coding frame structure prediction technology for downstream tasks at the coding end, significantly reducing the redundancy of the coding stream for downstream video analysis tasks and improving the downstream visual task performance of the decoded video at the same bit rate. At the same time, compared with the existing method that requires training multiple sets of codecs for multiple tasks separately, the present invention does not need to change the weights and decoding process of the codec, and is easier to apply and deploy.
[0073] By applying the method of this embodiment to data centers and mobile terminals in scenarios such as smart cities for codec model applications, in these scenarios, large-scale video data needs to be processed for downstream video analysis tasks, or the computing performance of the equipment is limited, and targeted codec models need to be trained. Through the technology described in this embodiment, the performance of the original codec in downstream visual tasks can be significantly improved, while reducing the training and application costs of the codec model.
[0074] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A visually controllable encoder-side deep learning video compression control method, characterized in that: include: Pre-train a deep learning codec optimized for human vision and freeze the model parameters to form the encoding end; Adding a coding mode prediction strategy to the coding end, performing training optimization for downstream tasks, and obtaining an optimized coding mode prediction model; The coding mode prediction model is trained, and after the training is completed, the coding frame structure prediction model is introduced, and the training is continued for downstream tasks to obtain a video codec that integrates coding mode prediction and coding frame structure prediction, and video decompression is implemented based on the video codec.
2. The visually controllable encoder-side deep learning video compression control method according to claim 1, characterized in that: The process of pre-training a deep learning codec optimized for human vision involves: The residual video codec is pre-trained separately, and the conditional video codec is pre-trained using the pre-trained model weights to obtain a deep learning codec optimized for human vision.
3. The visually controllable encoder-side deep learning video compression control method according to claim 1, characterized in that: The coding mode prediction strategy is added to the coding end, and training optimization is performed for downstream tasks. The process of obtaining an optimized coding mode prediction model includes: A coding mode prediction strategy is added to the encoding end, and multi-target tracking and video target detection tasks are selected for training respectively. Reasoning verification is performed on three video analysis tasks of multi-target tracking, video target detection and video action recognition to obtain an optimized coding mode prediction model.
4. The visually controllable encoder-side deep learning video compression control method according to claim 1, characterized in that: The loss function expression of the coding mode prediction model is: L m =R+λ m L t Among them, R is the bit rate required to encode the current frame, L t is the loss function size of the downstream task corresponding to the decoded frame, λ m A hyperparameter to balance the two.
5. The visually controllable encoder-side deep learning video compression control method according to claim 1, characterized in that: The process of training the coding mode prediction model includes: The coding mode prediction model uses the distribution parameter prior information of the coding feature estimation of the video encoder entropy model as input to the residual convolutional neural network; The residual convolutional neural network determines the importance of each position feature to the downstream machine vision task according to the distribution parameters of each position feature; For the feature map to be encoded, the residual convolutional neural network predicts a coding mode of "0" or "1" for each element in the feature map, and uses the Gumbel-Softmax sampling strategy to perform sampling according to the probability of the residual convolutional neural network output, and differentially selects a mode with a larger probability value as the coding mode to form a coding mode binary mask with the same shape as the feature map.
6. The visually controllable encoder-side deep learning video compression control method according to claim 5, characterized in that: For the feature map to be encoded, the process of the residual convolutional neural network predicting the encoding mode of "0" or "1" for each element in the feature map includes: For each element, if the prediction is "1", it means that the element is more important for the downstream visual task or it is difficult for the decoder to accurately predict the element based on the prior information of the entropy model, and the position needs to be encoded and transmitted; On the contrary, if the prediction is "0", it means that the downstream task is not sensitive to the element, or the decoding end can accurately predict the element based on the prior information of the entropy model, and there is no need to encode and transmit the position.
7. The visually controllable encoder-side deep learning video compression control method according to claim 1, characterized in that: The loss function expression of the coding frame structure prediction model is: in, The average bit rate required to encode the predicted frames in the current picture group, is the average loss function size of the downstream task corresponding to the decoded prediction frame of the picture group, λ g A hyperparameter to balance the two.
8. The visually controllable encoder-side deep learning video compression control method according to claim 1, characterized in that: The coding mode prediction model is trained, and after the training is completed, the coding frame structure prediction model is introduced. The process of continuing the training for downstream tasks includes: The frame structure prediction training at the encoding end is carried out in units of picture groups. Multi-target tracking and video target detection tasks are also selected for training respectively. Reasoning verification is carried out on three video analysis tasks, namely multi-target tracking, video target detection and video action recognition, to obtain a video codec that integrates coding mode prediction and coding frame structure prediction.
9. The visually controllable encoder-side deep learning video compression control method according to claim 8, characterized in that: The process of performing frame structure prediction training on the encoding side in units of picture groups includes: After obtaining the to-be-encoded picture group, firstly using the optical flow estimation network to perform inter-frame optical flow estimation based on the encoding frame structure prediction model to obtain the inter-frame optical flow; Subsequently, based on the coding frame structure prediction model, the YOLO detector is used to process the picture group frame by frame to generate a target area mask generated by the overlap of the detection frame, wherein the value of the area inside the frame is set to "1" and the value of the area outside the frame is set to "0". The target area mask of each predicted frame is multiplied by its corresponding inter-frame optical flow to obtain the optical flow information after the mask, and the optical flow information is input into the coding frame structure prediction model again, and the optical flow information and the inter-frame residual information are used as inputs, and finally a binary tensor with a length equal to the number of predicted frames is output, and the prediction value at each position is "1" on behalf of the coding type of the position predicted as P frame, and the prediction value is "0" on behalf of the coding type of the position predicted as P m frame; The coding frame structure prediction model also uses the Gumbel-Softmax sampling strategy to implement gradient calculation and back propagation during training.