Video Compression System and Method Based on Embedded Platform

Abstract

This invention discloses a video compression system and method based on an embedded platform, mainly addressing the problems of low compression efficiency, poor adaptability, and high computational complexity in existing technologies, which limit their application. The solution includes: 1) constructing a sample set using publicly available video data; 2) building and training a video compression model using modules for feature extraction, motion estimation, motion compression / decompression, motion compensation, residual compression / decompression, and feature reconstruction; 3) fine-tuning and lightweighting the trained model through residual block pruning; 4) converting the model to a format supported by the deployment platform; 5) decomposing the converted model into compression and decompression models, which are deployed to embedded platforms at both ends; 6) acquiring and preprocessing video data, then inputting it to the compression model at the encoding end to obtain a binary bitstream, which is transmitted to the decompression model at the decoding end to reconstruct the reconstructed frame. This invention features low model and computational complexity and can be deployed on embedded devices.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and further relates to image processing, specifically a video compression system and method based on an embedded platform. It can be deployed on side-end embedded devices to reduce the communication transmission time and bandwidth requirements of image files. Background Technology

[0002] With the continuous emergence of various new video application scenarios, people's requirements for video coding quality and efficiency are increasing, and traditional video coding technologies can no longer adequately meet these application needs. Over the past few decades, video compression technology has mainly focused on hybrid predictive / transform coding, and a series of video compression standards have been proposed for various fields such as video-on-demand and streaming media, including MPEG, H.264 / AVC, H.265 / HEVC, and AVS. These standards employ techniques such as motion estimation, intra-frame prediction, inter-frame coding, and entropy coding to optimize the rate-distortion performance of compression systems. However, they all utilize manually designed modules, and effective joint optimization between these modules is difficult, resulting in limited performance improvements.

[0003] In recent years, deep learning-based video compression algorithms have been extensively studied. Thanks to deep learning and end-to-end training strategies, deep learning-based video compression has surpassed traditional video compression technologies such as H.264 or H.265. Lu et al. proposed the first end-to-end video compression scheme, DVC, in their paper "DVC: An End-to-end Deep Video Compression Framework," which can be seen as a deep learning version of traditional video compression schemes. Specifically, their scheme uses an optical flow estimation module to obtain motion information between the current frame and the decoded frame. Motion compensation is also performed by the trained network to generate the prediction signal for the current frame. Two autoencoders are used to compress the prediction residual and motion information respectively. The entire network is jointly optimized using a single loss function, exhibiting flexibility and high performance. Later, Hu et al. proposed the FVC video compression scheme in "FVC: A New Framework towards Deep Video Compression in Feature Space," which, based on DVC, introduces video compression from the pixel domain to the feature domain, while using deformable convolution for motion compensation. Motion compensation or residual compression is performed in the feature space, and deformable convolutions based on dynamic kernels can better handle more complex non-rigid motion patterns between two consecutive frames, thereby improving motion compensation results. However, existing deep learning-based video compression methods are quite complex. These complex end-to-end video compression methods can only run on GPUs, and their millions of parameters make it impossible to deploy them on embedded side devices, severely limiting their application and promotion.

[0004] Existing traditional video compression algorithms suffer from low compression efficiency and poor adaptability due to the use of hand-designed modules and fixed algorithms; while end-to-end video compression algorithms based on deep learning are difficult to deploy on side-end embedded devices due to the large number of model parameters and computational load, thus limiting their application. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a video compression system and method based on an embedded platform. This addresses the problems of low compression efficiency, poor adaptability, and high computational complexity that limit the application of existing technologies. The invention aims to design a highly efficient and lightweight end-to-end video compression model and deploy it on a domestically developed embedded platform, thereby solving the aforementioned problems and further promoting the application of deep learning-based video compression models on side-end embedded devices.

[0006] The basic idea behind this invention is to construct and train an end-to-end video compression model, and then deploy the trained model to embedded platforms at both the compression and decompression ends. To improve compression efficiency on the embedded platforms, a method of skipping entropy coding is used to reduce entropy coding time, and residual block pruning is used to lightweight the model. This invention enables the deployment of deep learning-based video compression models on domestically developed embedded platforms, solving the problem that existing deep compression models are limited in application due to high model complexity, numerous parameters, and large computational load, and promoting the application of deep learning-based video compression models on side-end embedded devices.

[0007] The specific technical solution of the present invention to achieve the above objectives is as follows:

[0008] A video compression system based on an embedded platform includes a video acquisition module, a video compression module, a data storage module, a data transmission module, and a video decompression module; wherein,

[0009] The video acquisition module is used to acquire raw video information and upload it to the video compression module;

[0010] The video compression module consists of a feature extraction submodule, a motion estimation submodule, a motion compression submodule, a first motion decompression submodule, a first motion compensation submodule, a residual compression submodule, and a first residual decompression submodule. This module is deployed on a domestically produced NPU embedded device at the compression end and is used to compress the original video uploaded by the video acquisition module, obtain a motion vector binary file and a residual binary file, and output the file to the data storage module.

[0011] The data storage module is used to store the data generated by the video compression module;

[0012] The data transmission module is used to send data from the data storage module to the video decompression module over the communication link;

[0013] The video decompression module, deployed on a domestically produced NPU embedded device at the decompression end, consists of a second motion decompression submodule, a second motion compensation submodule, a second residual decompression submodule, and a feature reconstruction submodule. It is used to decode the input binary file and obtain the reconstructed video as the final output.

[0014] Furthermore, the aforementioned video acquisition module employs any hardware device capable of acquiring images, including mobile phones, professional cameras, monitoring equipment, and aerial drones; the data transmission module employs physical transmission devices, including coaxial cables and optical fibers; and both the data storage module and the data transmission module input and output binary files.

[0015] Furthermore, the aforementioned video compression module compresses the original video uploaded by the video acquisition module. Specifically, it first uses a feature extraction submodule to extract features from the current frame and the reference frame; then, a motion estimation submodule performs motion estimation on the features of the current frame and the reference frame in the feature domain to obtain motion vectors; the motion compression submodule encodes these vectors to obtain a binary motion vector file, and the first motion decompression submodule decodes this file to obtain the first reconstructed motion vector; the first reconstructed motion vector and the reference frame are then fed into a first motion compensation submodule for motion compensation to obtain the first predicted frame features; finally, the residual between the features of the current frame and the predicted frame is fed into a residual compression submodule to obtain a binary residual file, which is then fed into a first residual decompression submodule to obtain the first reconstructed residual features. The sum of the reconstructed residual features and the predicted frame features is used to update the reference frame features for compression of the next frame.

[0016] Furthermore, the aforementioned video decompression module decodes the input binary file. Specifically, the second motion decompression submodule decodes the input motion vector binary file into a second reconstructed motion vector. Then, the second reconstructed motion vector and the reference frame features are fed into the second motion compensation submodule for motion compensation to obtain the second predicted frame features. Next, the residual binary file is fed into the second residual decompression submodule to obtain the second reconstructed residual features. Finally, the reconstructed residual features are added to the predicted frame features to obtain the reconstructed frame features, which are simultaneously updated as the reference frame features for the next frame decompression. The feature reconstruction submodule then performs feature reconstruction on the reconstructed frame features to obtain the reconstructed frame.

[0017] A video compression method based on an embedded platform includes the following steps:

[0018] (1) Construct a sample set using publicly available video data and divide it into a training set and a test set; and preprocess the training set by setting the frame interval GOP.

[0019] (2) Construct an end-to-end video compression model based on deep learning. The specific steps are as follows:

[0020] (2.1) A feature extraction module is constructed using a convolutional layer and a residual layer to encode the input frame and the reference frame into the feature space and obtain the feature representation of the input frame and the reference frame;

[0021] (2.2) Construct a motion estimation module. In this module, the features of the input frame and the reference frame are first concatenated along the channel dimension, and then processed by two convolutional layers and activation functions to output the feature offset of motion estimation.

[0022] (2.3) Construct a motion compression / decompression module. The feature offset of motion estimation is used as the input of this module. The feature offset is extracted through motion transformation in the module to generate offset features. After quantization of the offset features, entropy encoding is performed. The compressed bitstream is obtained through entropy encoding, and the bitstream is entropy decoded to obtain the decoded offset features. The reconstructed feature offset is obtained through the inverse motion transformation network. The motion transformation network consists of three convolutional layers and two residual layers alternately. The entropy encoding is an improved entropy encoding network with an added skip entropy encoding prediction module. The skip entropy encoding prediction module consists of two convolutional layers, a Leaky ReLU activation function layer and a Gumbel Softmax / Max layer. It maps the input prior distribution parameters to a binary 0 / 1 mask to skip the encoding step.

[0023] (2.4) The reference frame features and motion offset are used as inputs to the deformable convolutional layer. Deformable convolution operation is used to perform motion compensation to obtain the distortion features. Then, the distortion features are refined through two convolutional layers to produce the final prediction frame.

[0024] (2.5) The network structure of residual compression is the same as that of motion compression and decompression, both of which include residual transformation, inverse residual transformation, entropy coding and entropy decoding modules; used to compress and decompress the residuals of the predicted frame features and the current frame features to obtain the reconstructed residual features;

[0025] (2.6) After adding the predicted frame and the reconstructed residual features, the reconstructed output frame is generated by the feature reconstruction module. The feature reconstruction module has the same network structure as the feature extraction module, that is, it is composed of a convolutional layer and a residual layer.

[0026] (2.7) The feature extraction module, motion estimation module, motion compression / decompression module, motion compensation module, residual compression / decompression module and feature reconstruction module constitute an end-to-end video compression model based on deep learning;

[0027] (3) Use the training set to train the video compression model constructed in step (2) until the model parameters converge. Save the model weight parameters and model structure that perform best on the test set to obtain the trained video compression model.

[0028] (4) Fine-tune and lightweight the trained model by residual block pruning:

[0029] (4.1) Scale the output of the residual block using a scaling factor;

[0030] (4.2) Add sparsity regularization to the scaling factor and optimize it by stochastically accelerating the proximal gradient APG;

[0031] A scaling factor β is introduced for each residual block in the network, and then a sparsity constraint is added to β during training to obtain a sparse β.

[0032] (4.3) Delete the residual block structure where β is 0 in all residual blocks to complete the lightweighting process;

[0033] (5) Perform an equivalent conversion on the video compression model to convert it into a model format supported by the AI ​​hardware platform during the deployment phase;

[0034] (6) Decompose the equivalent transformation model into two parts: a compressed model and a decompressed model. Use custom symbolic functions to map the deformable convolution operations in the two parts of the model to the corresponding ONNX operator operations. Save the equivalent transformation model structure and parameters as ONNX operator inference architecture model files. Then, use the model conversion tool provided by the computing platform to convert the ONNX operator inference architecture model files into deployment model files (OM). Deploy the OMs of the two parts of the model to the embedded platforms at both ends.

[0035] (7) Acquire video and extract video frames one by one for preprocessing. Input the preprocessed video into the encoding end of the trained video compression model, compress it into a bitstream and then transmit it to the storage module for storage. Finally, the bitstream is transmitted from the storage module to the decoding end through the data transmission module for decompression to obtain the reconstructed video frames.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] First, considering the large number of residual blocks in the compressed network model, this invention scales and prunes the residual blocks during training to construct a more lightweight and compact network model. Specifically, it uses a parameter called scaling factor to scale the output of the residual blocks, adds sparsity regularization to these factors, and solves the optimization problem through an improved stochastic accelerated proximal gradient (APG) method. By forcing certain factors to zero, the corresponding structures are safely deleted, completing the pruning of unimportant parts of the network and effectively reducing the number of network parameters and computational cost.

[0038] Secondly, this invention takes into account that most elements in the feature map have been accurately predicted by the super-prior network, and therefore uses a skip entropy coding method in the motion and residual entropy coding parts, skipping most of the accurately predicted features and only performing entropy coding on a few inaccurately predicted features, thus shortening the time of entropy coding and entropy decoding; it can significantly improve the efficiency of entropy coding and shorten the time of entropy coding and entropy decoding.

[0039] Third, this invention solves the problem of the difficulty in deploying deep learning-based video compression algorithms. By optimizing entropy coding and lightweight models, it accelerates the speed of video compression on the neural network processing unit (NPU). At the same time, by using a custom deformable convolution operator to derive the model, it can be deployed and run on the Huawei Ascend Atlas 200 IDK A2 embedded platform. Attached Figure Description

[0040] Figure 1 This is a diagram showing the overall framework of the video compression system constructed in this invention.

[0041] Figure 2 This is a flowchart illustrating the implementation of the method of the present invention;

[0042] Figure 3 This is a flowchart illustrating the workflow of the video compression system in this invention.

[0043] Figure 4 This is a schematic diagram of the video compression algorithm structure used in this invention;

[0044] Figure 5 This is a schematic diagram of the network structure of the motion and residual compression module in this invention;

[0045] Figure 6 This is a schematic diagram of the structure of the skip entropy coding module in this invention;

[0046] Figure 7 This is a schematic diagram of the motion compensation module in this invention.

[0047] Figure 8 This is a schematic diagram of the residual block pruning process in this invention;

[0048] Figure 9 This is a flowchart illustrating the implementation of the custom operator in this invention;

[0049] Figure 10 This is a comparison chart of simulation test results provided in the embodiments of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described clearly and completely below with reference to the accompanying drawings.

[0051] Example 1: Refer to Appendix Figure 1This invention proposes a video compression system based on an embedded platform, comprising a video acquisition module, a video compression module, a data storage module, a data transmission module, and a video decompression module. The video acquisition module utilizes any hardware device capable of acquiring images, such as a mobile phone, professional camera, surveillance equipment, or aerial drone, to acquire raw video information and upload it to the video compression module. The video compression module consists of a feature extraction submodule, a motion estimation submodule, a motion compression submodule, a first motion decompression submodule, a first motion compensation submodule, a residual compression submodule, and a first residual decompression submodule. This module is deployed on a domestically produced NPU embedded device at the compression end and is used to compress the raw video uploaded by the video acquisition module. The system acquires a motion vector binary file and a residual binary file, and outputs these files to the data storage module. The data storage module stores the data generated by the video compression module. The data transmission module uses physical transmission equipment, such as coaxial cable or optical fiber, to send the data from the data storage module to the video decompression module over a communication link. In this embodiment, both the data storage module and the data transmission module input and output binary files. The video decompression module, deployed on a domestically produced NPU embedded device at the decompression end, consists of a second motion decompression submodule, a second motion compensation submodule, a second residual decompression submodule, and a feature reconstruction submodule. It decodes the input binary file to obtain the reconstructed video as the final output.

[0052] The aforementioned video compression module compresses the raw video uploaded by the video acquisition module. Specifically, it first extracts features from the current frame and the reference frame using a feature extraction submodule. Then, it estimates motion vectors from the features of the current frame and the reference frame in the feature domain using a motion estimation submodule. The motion compression submodule then encodes these vectors to obtain a binary motion vector file, which is then decoded by a first motion decompression submodule to obtain the first reconstructed motion vector. The first reconstructed motion vector and the reference frame are then fed into a first motion compensation submodule for motion compensation to obtain the first predicted frame features. Finally, the residual between the features of the current frame and the first predicted frame is fed into a residual compression submodule to obtain a binary residual file, which is then fed into a first residual decompression submodule to obtain the first reconstructed residual features. The sum of the first reconstructed residual features and the predicted frame features is used to update the reference frame features for compression of the next frame.

[0053] The aforementioned video decompression module decodes the input binary file as follows: First, the second motion decompression submodule decodes the input motion vector binary file into a second reconstructed motion vector. Then, the second reconstructed motion vector and the reference frame features are sent to the second motion compensation submodule for motion compensation to obtain the second predicted frame features. Next, the residual binary file is sent to the second residual decompression submodule to obtain the second reconstructed residual features. Finally, the second reconstructed residual features are added to the predicted frame features to obtain the reconstructed frame features, which are then updated as the reference frame features for the next frame decompression. The feature reconstruction submodule then performs feature reconstruction on the reconstructed frame features to obtain the reconstructed frame.

[0054] Example 2: Refer to Appendix Figure 2 The present invention proposes a video compression method based on an embedded platform, which specifically includes the following steps:

[0055] Step 1. Construct a sample set using publicly available video data, dividing it into a training set and a test set. Preprocess the training set by setting a frame interval (GOP). In this embodiment, the preprocessing includes: performing single-frame training on the video data in the training set during the initial training phase, i.e., using two frames as a group for each video frame compression; and performing multi-frame training in the later training phase, i.e., using N images as a group, with the first image of each group as the keyframe (i.e., the starting reference frame), and continuously performing N-1 video frame compressions. During this period, each reconstructed frame is used as the reference frame for the next frame compression.

[0056] Step 2. Construct an end-to-end video compression model based on deep learning. The specific steps are as follows:

[0057] (2.1) A feature extraction module is constructed using a convolutional layer and a residual layer to encode the input frame and the reference frame into the feature space and obtain the feature representation of the input frame and the reference frame;

[0058] (2.2) Construct a motion estimation module. In this module, the features of the input frame and the reference frame are first concatenated along the channel dimension, and then processed by two convolutional layers and activation functions to output the feature offset of motion estimation.

[0059] (2.3) Construct a motion compression / decompression module. The feature offset of motion estimation is used as the input of this module. The feature offset is extracted by the transformation network in the module to generate offset features. The offset features are quantized and then entropy encoded. The compressed bitstream is obtained through entropy encoding, and the bitstream is entropy decoded to obtain the decoded offset features. The reconstructed feature offset is obtained through the inverse transformation network. The motion transformation network consists of three convolutional layers and two residual layers alternately. The inverse transformation network consists of three deconvolutional layers and two residual layers alternately. The entropy encoding is an improved entropy encoding network with an added skip entropy encoding prediction module. The skip entropy encoding prediction module consists of two convolutional layers, a Leaky ReLU activation function layer and a Gumbel Softmax / Max, which maps the input prior distribution parameters to a binary 0 / 1 mask to skip the encoding step.

[0060] The improved entropy coding network only needs to encode the prior information at the position corresponding to the mask information of 1 during entropy coding, and skips the encoding of the prior information at the position corresponding to the mask information of 0. Similarly, during entropy decoding, the pixels at the position corresponding to the mask information of 0 do not need to be entropy decoded, and the mean value is directly used to replace the value of the corresponding pixel. Only the prior information at the position corresponding to the mask information of 1 needs to be decoded.

[0061] Regarding the entropy coding optimization method, C2F (Coarse-to-Fine) can also be considered. A super-prior guided mode prediction network is proposed to predict the skipping mode of effective residual coding. The skipped elements are set to zero, which can also achieve the fast entropy coding purpose achieved by adding a skip entropy coding prediction module in this embodiment.

[0062] (2.4) The reference frame features and motion offset are used as inputs to the deformable convolutional layer. Deformable convolution operation is used to perform motion compensation to obtain the distortion features. Then, the distortion features are refined through two convolutional layers to produce the final prediction frame.

[0063] (2.5) The network structure of the residual compression / decompression module is the same as that of the motion compression / decompression module, both including transform, inverse transform, quantization, entropy coding and entropy decoding; used to compress and decompress the residual between the predicted frame features and the current frame features to obtain the reconstructed residual features;

[0064] In this embodiment, the residual information is first extracted using a transform network to generate residual features. Then, the residual features are entropy encoded, and an improved entropy encoding network is used to accelerate the entropy encoding process. The bitstream obtained by entropy encoding is then entropy decoded to obtain the decoded residual features, which are then fed into the residual inverse transform network to obtain the reconstructed residual features.

[0065] (2.6) After adding the predicted frame and the reconstructed residual features, the reconstructed output frame is generated by the feature reconstruction module. The feature reconstruction module has the same network structure as the feature extraction module, that is, it consists of a convolutional layer and a residual layer.

[0066] (2.7) The end-to-end video compression model based on deep learning consists of a feature extraction module, a motion estimation module, a motion compression / decompression module, a motion compensation module, a residual compression / decompression module, and a feature reconstruction module.

[0067] Step 3. Train the video compression model constructed in Step 2 using the training set until the model parameters converge. Save the optimal model weights and structure from the test set to obtain the trained video compression model. In this embodiment, the training process uses the following loss function:

[0068]

[0069] Among them, R o and R r These represent the number of bits used to encode the motion offset and the residual, respectively. Indicates input frame X t and reconstructed frames The mean square error between the two values ​​is λ, which is a hyperparameter used to control the bit rate-distortion tradeoff.

[0070] Step 4. Fine-tune and lightweight the trained model by residual block pruning. The steps are as follows:

[0071] (4.1) Introduce a scaling factor β for each residual block in the network, add sparsity constraints to β during training, and optimize by randomly accelerating the proximal gradient to obtain a sparse β;

[0072] (4.2) When β is less than the threshold, set it to 0 and delete all residual blocks in which β is 0 to complete the lightweighting process;

[0073] The residual block pruning is optimized during training according to the following formula:

[0074] r i+1 =r i +β i F i (r i W i ),

[0075] Where, r i and r i+1 F is the input and output of the i-th residual block. i It is the residual function, W iβ is the weight parameter of the i-th residual block; i It is the scaling factor corresponding to the i-th residual block. Sparse β is obtained by adding sparsity constraints to β during training. When β... i When the value is less than the threshold, the value is set to 0, which allows for the safe deletion of the i-th residual block, as its output does not contribute to subsequent calculations, and this residual block is skipped during inference. In this embodiment, lightweight processing can also be implemented using channel pruning or other block pruning methods.

[0076] Step 5. Perform an equivalent transformation on the video compression model to convert it into a model format supported by the AI ​​hardware platform during the deployment phase;

[0077] Step 6. Decompose the equivalently transformed model into two parts: a compressed model and a decompressed model. ONNX provides an open-source format for AI models in deep learning and traditional ML. It defines an extensible computation graph model, as well as built-in operators and standard data types. Custom symbolic functions are used to map deformable convolution operations in both parts of the model to corresponding ONNX operator operations. The equivalently transformed model structure and parameters are saved as ONNX operator inference architecture model files. Then, the model conversion tool provided by the computing platform is used to convert the ONNX operator inference architecture model files into deployment model files (OMs). The OMs of both parts of the model are then deployed to the embedded platforms at both ends.

[0078] Step 7. Acquire video and extract video frames one by one for preprocessing. Input the preprocessed video into the encoding end of the trained video compression model, compress it into a bitstream, and then send it to the storage module for storage. Finally, the bitstream is sent from the storage module to the decoding end through the data transmission module for decompression to obtain the reconstructed video frames.

[0079] Example 3: The overall implementation steps of the video compression method proposed in this example are the same as in Example 2. Now, in conjunction with the appendix... Figure 2-10 The implementation process of this invention will be further described in detail with specific examples:

[0080] like Figure 2As shown, the method includes the following steps: S1. Prepare a publicly available video dataset as both a training set and a test set. S2. Preprocess the training set and set the frame interval (GOP, Group of Pictures). S3. Construct an end-to-end video compression model based on deep learning. S4. Train the end-to-end video compression model using the training set and save the optimal model weight parameters and model structure on the test set. S5. After model training, fine-tune the model and perform lightweight processing to reduce the computational complexity and improve inference speed. S6. Perform an equivalent transformation on the video compression model to convert it to a model format supported by the AI ​​hardware platform during deployment. S7. Decompose the converted model and deploy the compression model and decompression model to the Ascend Atlas 200IDK A2 embedded platforms at both ends. S8. Acquire video and extract video frames frame by frame for preprocessing. Input the preprocessing data into the encoding end of the video compression model, compress it into a bitstream, and then input it into the decoding end for decompression to obtain the reconstructed video frames. The specific implementation steps are as follows:

[0081] Step S1: The dataset used is The Vimeo-90k dataset, which is an 82GB video compression dataset released by Vimeo. This dataset contains more than 90,000 video clips with a resolution of 448×256.

[0082] Step S2: Preprocess the training set video. In the early stages of training, single-frame training is performed, that is, two frames are used as a group for video frame compression each time. Starting from the 2 millionth batch, multi-frame training is performed, that is, 12 images are used as a group, and the first image of each group is used as the keyframe (starting reference frame) for 11 consecutive video frame compressions. During this period, the reconstructed frame is used as the reference frame for the next frame compression.

[0083] Step S3: Construct an end-to-end video compression model. During the training phase, the model is as follows: Figure 4 As shown, it includes modules for feature extraction, motion estimation, motion compression and decompression, motion compensation, and residual compression and decompression. The specific steps are as follows:

[0084] S31: Feature Extraction

[0085] The feature extraction module uses a convolutional layer and a residual layer to encode the input frame and the reference frame into the feature space.

[0086] S32: Motion Estimation

[0087] After obtaining the feature representations of the input frame and the reference frame, they are input into the motion estimation submodule to obtain the estimated feature offset M. tIn the motion estimation module, the two input features are first concatenated along the channel dimension, and then passed through two convolutional layers and an activation function to obtain the feature offset M. t .

[0088] S33: Motion compression and motion decompression

[0089] Feature offset M t Send in Figure 5 Compression and decompression are performed in the motion compression model. For example... Figure 5 When using a transform network for M t Further offset features are extracted to generate the transformed feature y. The transformation network consists of three alternating convolutional layers and two residual layers. After quantization, y is entropy encoded. Before entropy encoding, y first passes through a super-prior encoder module to further extract and compress statistical information in y, generating a more compact super-prior feature z. Then, the super-prior feature z is quantized, entropy encoded, and entropy decoded to obtain the quantized super-prior feature. The quantized prior features are processed through a network module. The prior distribution parameters (μ, σ) are obtained through mapping. Using the prior distribution N(μ, σ), the quantized y is entropy-encoded to obtain a binary bitstream. The binary bitstream is then entropy-decoded to obtain... The result is obtained by feeding it into the inverse transform network. Right now Figure 4 In

[0090] Entropy coding can only be processed serially on a CPU and cannot be accelerated on an NPU. To further accelerate the entropy coding process, this invention designs a skip-entropy coding method for faster entropy coding of the quantized y. The structure of skip-entropy coding is as follows: Figure 6 As shown, the skip-encoding prediction module first generates 0 / 1 mask information for each feature in the quantized y. This mask is used to mask the features before entropy encoding of the prior information. A 1 in the mask indicates that the predicted prior information is uncertain, requiring entropy encoding for the corresponding pixel position; a 0 indicates that the prior information has been well predicted, and the corresponding pixel position does not need entropy encoding, allowing the mean value to be used directly. During entropy encoding, only the prior information at positions with a mask value of 1 needs to be encoded, while the prior information at positions with a mask value of 0 is skipped, significantly reducing encoding time. Similarly, during entropy decoding, pixels with a mask value of 0 do not need entropy decoding; the mean value is used directly. Decoding only the prior information at positions with a mask value of 1 also reduces decoding time.

[0091] S34: Motion Compensation

[0092] This invention uses deformable convolution operations for motion compensation to generate predicted features. The network structure is as follows: Figure 7 As shown, during motion compensation, the deformable convolutional layer performs deformable convolution operations using reference frame features and motion offsets as input. In this way, the learned dynamic offset kernel can be used to better compress videos with complex non-rigid motion patterns and produce more accurate distortion features. To generate more accurate prediction features, two convolutional layers are used to further refine the output features of the deformable convolutional layer, ultimately producing the final prediction frame.

[0093] S35: Residual Compression and Decompression

[0094] The residuals between the predicted frame features and the current frame features are compressed and decompressed to obtain the reconstructed residual features. The network structure for residual compression and... Figure 5 The network structures for motion compression and decompression are exactly the same, including modules for transform, inverse transform, quantization, entropy coding, and entropy decoding.

[0095] S36: Feature Reconstruction

[0096] Finally, the residual information will be reconstructed. Add back the predicted frame The reconstructed output frame is generated through a feature reconstruction module. The feature reconstruction module has the same network structure as the feature extraction module, using one convolutional layer and one residual layer.

[0097] Step S4: Training Details

[0098] During model training, our loss function is a bitrate-distortion tradeoff:

[0099]

[0100] Where R o and R r These represent the number of bits used to encode the motion offset and the residual, respectively. Indicates input frame X t and reconstructed frames The distortion between the two images is denoted by d(), where d() represents the mean squared error or MS-SSIM (used to measure the perceptual similarity between two images), and λ is a hyperparameter used to control the bitrate-distortion tradeoff. The hyperparameters λ of the trained model were set to 512, 1024, 2048, and 8192, with different λ corresponding to different bitrates; the larger the λ, the higher the bitrate. A total of 2.3 million training batches were used. To improve the model's generalization ability and prevent it from getting stuck in local optima during training, and to quickly converge to the global optimum, this experiment adopted a learning rate decay strategy, with an initial learning rate set to 5e-5, decaying starting from the 1.9 millionth training batch.

[0101] Step S5: Model Lightweighting

[0102] The presence of numerous residual blocks in video compression models can negatively impact inference speed during practical deployment. This invention addresses this by employing Residual Block Pruning to accelerate inference and reduce the number of model parameters during fine-tuning of the trained model. Specifically, a parameter called a scaling factor is used to scale the output of the residual blocks. Sparse regularization is then applied to these parameters, and this optimization problem is addressed using an improved Stochastic Accelerated Proximal Gradient (APG) method.

[0103] A scaling factor β is introduced for each residual block in the network to scale this specific structure of the residual block. Then, during training, a sparsity constraint is added to β, i.e., L1 regularization is applied, with the goal of obtaining a sparse β, as shown in the following formula:

[0104] r i+1 =r i +β i F i (r i W i )

[0105] Where r i and r i+1 It is the input and output of the i-th block, F i It is the residual function, W i This is a parameter of the block. β i It is the scaling factor corresponding to the i-th residual block.

[0106] When β i When the value is less than the threshold, it is set to 0. Its output does not contribute to subsequent calculations, meaning the i-th residual block can be safely deleted. During inference, this residual will be skipped directly. Figure 8 Medium gray blocks. Figure 8 The structure of residual pruning is explained.

[0107] Step S6: Export the ONNX operator inference architecture model

[0108] After training, the weights are saved and exported as an ONNX operator inference architecture model. Since our model uses deformable convolutions, but the latest version of PyTorch does not support direct conversion of networks with meaningful deformable convolutions into ONNX operator inference architecture models, a custom operator is used to implement the functionality of deformable convolutions.

[0109] like Figure 9As shown, we need to define a custom symbolic function. In this function, we map deformable convolution operations in PyTorch to corresponding ONNX operator operations. PyTorch will call the symbolic function to generate an ONNX operator graph structure that matches the logic of the corresponding operation. Next, we register this custom symbolic function using `register_custom_op_symbolic`, associating it with the target operator. Then, we use `torch.onnx.export` to export the ONNX operator inference architecture model containing the custom operator. Finally, we need to verify that the exported model runs correctly to ensure that its functionality is consistent with the original model.

[0110] Due to the specific hardware architecture of the Ascend Atlas200I AI computing platform, it cannot directly accelerate inference of network models. Therefore, a tool is needed for model deployment conversion. First, the equivalent model structure and parameters are saved as ONNX inference architecture model files. Then, the model conversion tool ATC provided by the Ascend Atlas200I AI computing platform is used to convert the ONNX inference architecture model files into OM deployment model files. Finally, the OM model files are deployed for inference applications.

[0111] Step S7: Deployment of the Atlas200I platform

[0112] The split models are deployed to the video compression and decompression ends respectively. In this embodiment, both the video compression and decompression ends use the Huawei Ascend Atlas 200 IDK A2 platform. Figure 1 This shows our deployed video compression system. The model is deployed to both ends of the Ascend Atlas 200IDK A2 embedded platform, as shown in the following implementation:

[0113] (6.1) Use custom symbolic functions to map deformable convolution operations in the model to corresponding ONNX operators, and save the equivalent transformation model structure and parameters as an ONNX inference architecture model file;

[0114] (6.2) Use the model conversion tool ATC provided by the Ascend Atlas200I AI computing platform to convert the ONNX model file into an OM deployment model file;

[0115] (6.3) Deploy the OM model files of the compressed and decompressed ends to the Ascend Atlas 200IDK A2 embedded platforms at both ends;

[0116] Step S8: Reasoning Application

[0117] The video acquisition device captures video and extracts video frames one by one for preprocessing such as image cropping. The extracted frames are then fed into the video compression module. After compression, the bitstream is transmitted to the storage module for storage. Finally, the bitstream is transferred from the storage module to the video decompression module via the data transmission module for decompression. The workflow diagram is as follows: Figure 3 As shown.

[0118] The effects of the present invention will be further explained below with reference to simulation experiments.

[0119] 1. Simulation conditions:

[0120] The simulation experiments of this invention were conducted in the hardware environment of Huawei Ascend Atlas 200IDK A2 platform and the software environment of Linux operating system.

[0121] 2. Simulation Content

[0122] The video compression model constructed in this invention is compared with the benchmark models (DVC, FVC, H.265) on the test dataset in terms of rate-distortion performance and encoding / decoding time before and after model optimization.

[0123] 3. Simulation Results

[0124] The PSNR and MS-SSIM metrics were tested on the UVG and MCL_JCV datasets respectively. The test results were compared to... Figure 10 As shown. The compression model of this invention, based on FVC, has been lightweighted and deployed on an embedded platform. Its performance in both PSNR and MS-SSIM is slightly lower than that of FVC, but it is still significantly better than traditional H.265 and deep learning-based DVC.

[0125] Using the UVG dataset as the test set, with a training hyperparameter λ of 2048 and a video resolution of 1080p, the average compression and decompression time per frame before and after optimization is shown in the table below:

[0126] Model Model before optimization Skip Entropy Coding Lightweight Optimized model Overall acceleration ratio Compression time 1006ms 706ms 878ms 578ms 1.74 Decompression time 698ms 469ms 597ms 368ms 1.89

[0127] Before optimization, the compression time of the compression model in this invention was 1006 ms, and the decompression time was 698 ms. In the ablation experiment, the compression time of the model with only skipped entropy coding was 706 ms, and the decompression time was 469 ms. With only lightweighting of the model, the compression time was 878 ms, and the decompression time was 597 ms. With both skipped entropy coding and lightweighting added, the compression time was 578 ms, and the decompression time was 368 ms. Compared to the model before optimization, the compression speedup of this invention is 1.74, and the decompression speedup is 1.89. Under the optimization of this invention, both compression and decompression times achieve good speedups.

[0128] The above simulation analysis proves the correctness and effectiveness of the method proposed in this invention.

[0129] The parts of this invention not described in detail are common knowledge to those skilled in the art.

[0130] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Obviously, those skilled in the art, after understanding the content and principle of the present invention, may make various modifications and changes in form and detail without departing from the principle and structure of the present invention. However, these modifications and changes based on the concept of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A video compression method based on an embedded platform, characterized in that, Includes the following steps: (1) Construct a sample set using publicly available video data, and divide it into a training set and a test set; and preprocess the training set by setting the frame interval GOP. (2) Construct an end-to-end video compression model based on deep learning. The specific steps are as follows: (2.1) A feature extraction module is constructed using a convolutional layer and a residual layer to encode the input frame and the reference frame into the feature space and obtain the feature representation of the input frame and the reference frame; (2.2) Construct a motion estimation module. In this module, the features of the input frame and the reference frame are first concatenated along the channel dimension, and then processed by two convolutional layers and activation functions to output the feature offset of motion estimation. (2.3) Construct a motion compression / decompression module, take the motion estimation feature offset as the input of the module, extract the feature offset through the transformation network in the module to generate offset features, quantize the offset features and entropy encode to obtain the compressed bitstream, entropy decode the bitstream to obtain the decoded offset features, and then pass through the inverse transformation network to obtain the reconstructed feature offset. The transform network consists of three convolutional layers and two residual layers alternatingly; the inverse transform network consists of three deconvolutional layers and two residual layers alternatingly; the entropy coding is an improved entropy coding network with an added skip entropy coding prediction module, wherein the skip entropy coding prediction module consists of two convolutional layers, a Leaky ReLU activation function layer, and a GumbelSoftmax / Max layer, which maps the input prior distribution parameters to a binary 0 / 1 mask for skipping the coding step; (2.4) Using the reference frame features and motion offset as input to the deformable convolutional layer, the deformable convolution operation is used to perform motion compensation to obtain the distortion features, and then the distortion features are refined through two convolutional layers to generate the final prediction frame. (2.5) The network structure of the residual compression / decompression module is the same as that of the motion compression / decompression module, both including transform, inverse transform, quantization, entropy coding and entropy decoding; used to compress and decompress the residuals of the predicted frame features and the current frame features to obtain the reconstructed residual features; (2.6) After adding the predicted frame and the reconstructed residual features, the reconstructed output frame is generated by the feature reconstruction module. The feature reconstruction module has the same network structure as the feature extraction module, that is, it is composed of a convolutional layer and a residual layer. (2.7) The end-to-end video compression model based on deep learning consists of a feature extraction module, a motion estimation module, a motion compression / decompression module, a motion compensation module, a residual compression / decompression module, and a feature reconstruction module. (3) Use the training set to train the video compression model constructed in step (2) until the model parameters converge, save the model weight parameters and model structure that perform best on the test set, and obtain the trained video compression model. (4) Fine-tune and lightweight the trained model by residual block pruning: (4.1) Introduce a scaling factor for each residual block in the network. During the training process By adding sparsity constraints and optimizing by stochastically accelerating the proximal gradient, sparseness can be obtained. ; (4.2) When If the value is less than the threshold, set it to 0, and apply this to all residual blocks. Delete residual blocks with a value of 0 to complete the lightweighting process; (5) Perform an equivalent conversion on the video compression model to convert it into a model format supported by the AI ​​hardware platform during the deployment phase; (6) Decompose the equivalent transformation model into two parts: a compressed model and a decompressed model. Use custom symbolic functions to map the deformable convolution operations in the two parts of the model to the corresponding ONNX operator operations. Save the equivalent transformation model structure and parameters as ONNX operator inference architecture model files. Then use the model conversion tool provided by the computing platform to convert the ONNX operator inference architecture model files into deployment model files (OM). Deploy the OMs of the two parts of the model to the embedded platforms at both ends. (7) Acquire video and extract video frames one by one for preprocessing. Input the preprocessed video into the encoding end of the trained video compression model, compress it into a bitstream and then transmit it to the storage module for storage. Finally, the bitstream is transmitted from the storage module to the decoding end through the data transmission module for decompression to obtain the reconstructed video frames.

2. The method according to claim 1, characterized in that: The preprocessing of setting the frame interval GOP in step (1) specifically involves the video data in the training set. In the early stage of training, single-frame training is performed, that is, two frames are used as a group for video frame compression each time. In the later stage of training, multi-frame training is performed, that is, N images are used as a group, and the first image of each group is used as the key frame, i.e. the starting reference frame. Video frame compression is performed continuously N-1 times. During this period, the reconstructed frame is used as the reference frame for the next frame compression.

3. The method according to claim 1, characterized in that: The training described in step (3) is performed using the following loss function: ； in, and These represent the number of bits used to encode the motion offset and the residual, respectively. Indicates input frame and reconstructed frames The mean square error between them It is a hyperparameter used to control the bitrate-distortion tradeoff.

4. The method according to claim 1, characterized in that: The residual block pruning described in step (4) is performed according to the following formula: ； in, and These are the inputs and outputs of the i-th residual block. It is the residual function. These are the weight parameters of the i-th residual block; It is the scaling factor corresponding to the i-th residual block.

5. The method according to claim 1, characterized in that: In the improved entropy coding network described in step (2.3), during entropy coding, only the prior information at the corresponding position where the mask information is 1 needs to be encoded, and the prior information at the corresponding position where the mask information is 0 is not encoded, i.e., the encoding is skipped; during entropy decoding, the pixels at the corresponding position where the mask information is 0 do not need to be entropy decoded, and the mean value is directly used to replace the value of the corresponding pixel point, and only the prior information at the corresponding position where the mask information is 1 needs to be decoded.

6. The method according to claim 1, characterized in that: The reconstructed residual features described in step (2.5) are first extracted by residual transformation to generate residual features; then, the residual features are entropy encoded, and an improved entropy encoding network is used to accelerate the entropy encoding process; the bitstream obtained by entropy encoding is entropy decoded to obtain the decoded residual features, which are then fed into the residual inverse transform network to obtain the reconstructed residual features.

7. A video compression system for implementing the method of claim 1, characterized in that, include: The system includes a video capture module, a video compression module, a data storage module, a data transmission module, and a video decompression module. The video acquisition module is used to acquire raw video information and upload it to the video compression module; The video compression module consists of a feature extraction submodule, a motion estimation submodule, a motion compression submodule, a first motion decompression submodule, a first motion compensation submodule, a residual compression submodule, and a first residual decompression submodule. This module is deployed on the NPU embedded device at the compression end and is used to compress the original video uploaded by the video acquisition module, obtain a motion vector binary file and a residual binary file, and output the file to the data storage module. The data storage module is used to store the data generated by the video compression module; The data transmission module is used to send data from the data storage module to the video decompression module over the communication link; The video decompression module, deployed on the NPU embedded device at the decompression end, consists of a second motion decompression submodule, a second motion compensation submodule, a second residual decompression submodule, and a feature reconstruction submodule. It is used to decode the input binary file and obtain the reconstructed video as the final output.

8. The video compression system according to claim 7, characterized in that: The video acquisition module uses any hardware device capable of acquiring images, including mobile phones, professional cameras, monitoring equipment, and aerial drones; the data transmission module uses physical transmission devices, including coaxial cables and optical fibers; both the data storage module and the data transmission module input and output binary files.

9. The video compression system according to claim 7, characterized in that: The video compression module compresses the raw video uploaded by the video acquisition module. Specifically, it first extracts features from the current frame and the reference frame using a feature extraction submodule. Then, it estimates motion vectors from the features of the current frame and the reference frame in the feature domain using a motion estimation submodule. The motion compression submodule then encodes these vectors to obtain a binary motion vector file, which is then decoded by a first motion decompression submodule to obtain a first reconstructed motion vector. The first reconstructed motion vector and the reference frame are then fed into a first motion compensation submodule for motion compensation to obtain a first predicted frame feature. Finally, the residual between the features of the current frame and the first predicted frame is fed into a residual compression submodule to obtain a residual binary file, which is then fed into a first residual decompression submodule to obtain a first reconstructed residual feature. The sum of the first reconstructed residual feature and the first predicted frame feature is used to update the reference frame feature for compression of the next frame.

10. The video compression system according to claim 8, characterized in that: The video decompression module decodes the input binary file as follows: First, the second motion decompression submodule decodes the input motion vector binary file into a second reconstructed motion vector. Then, the second reconstructed motion vector and the reference frame features are sent to the second motion compensation submodule for motion compensation to obtain the second predicted frame features. Next, the residual binary file is sent to the second residual decompression submodule to obtain the second reconstructed residual features. Finally, the second reconstructed residual features are added to the second predicted frame features to obtain the reconstructed frame features, which are then updated as the reference frame features for the next frame decompression. The feature reconstruction submodule then performs feature reconstruction on the reconstructed frame features to obtain the reconstructed frame.

Images