Video compression system and method based on embedded platform

Abstract

The invention discloses a video compression system and method based on an embedded platform, and mainly solves the problem of application limitation caused by low compression efficiency, poor adaptability and large calculation amount in the prior art. The scheme comprises the following steps: 1) constructing a sample set by using public video data; 2) constructing and training a video compression model by a feature extraction module, a motion estimation module, a motion compression / decompression module, a motion compensation module, a residual error compression / decompression module and a feature reconstruction module; 3) performing fine tuning and lightweight processing on the trained model through residual block pruning; 4) converting the model into a model format supported by a deployment platform; 5) decomposing the converted model into a compression model and a decompression model, and respectively deploying the compression model and the decompression model into embedded platforms at two ends; and 6) collecting a video, preprocessing the video, inputting the preprocessed video into the compression model of the coding end to obtain a binary code stream, and transmitting the binary code stream to the decompression model of the decoding end for reconstruction to obtain a reconstructed frame. The method is low in model complexity and calculation complexity, and can be deployed on embedded equipment.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and further relates to image processing, and is specifically a video compression system and method based on an embedded platform, which can be deployed on a side embedded device to reduce the communication transmission time and bandwidth requirements of image files. Background Art

[0002] With the continuous emergence of various emerging video application scenarios, people's requirements for video coding quality and efficiency are increasing, and traditional video coding technology can no longer meet application needs well. In the past few decades, video compression technology has mainly focused on hybrid prediction / transform coding, and a series of video compression standards have been proposed for various fields such as video on demand and streaming media, including MPEG, H.264 / AVC, H.265 / HEVC and AVS. They use motion estimation, intra-frame prediction, inter-frame coding, entropy coding and other technologies to optimize the rate-distortion performance of the compression system, but they are all composed of manually designed modules, and it is difficult to effectively jointly optimize the modules, resulting in limited performance improvement.

[0003] In recent years, deep learning-based video compression algorithms have been widely studied. Thanks to deep learning and end-to-end training strategies, deep learning-based video compression has surpassed traditional video compression technologies such as H.264 or H.265. Lu et al. proposed the first end-to-end video compression scheme DVC in DVC: An End-to-end Deep Video Compression Framework, which can be regarded as a deep learning version of the traditional video compression scheme. Specifically, in their scheme, an optical flow estimation module is used to obtain the motion information between the current frame and the decoded frame. Motion compensation is also performed by a trained network to generate a prediction signal for the current frame. Two autoencoders are used to compress the prediction residual and motion information respectively. The entire network is jointly optimized using a single loss function, which is flexible and has high performance. Later, Hu et al. proposed the FVC video compression scheme in FVC: A New Framework towards Deep Video Compression in Feature Space, which introduced video compression from the pixel domain to the feature domain based on DVC, and used deformable convolution for motion compensation. Motion compensation or residual compression is performed in the feature space. At the same time, by using deformable convolution based on dynamic kernels, more complex non-rigid motion patterns between two consecutive frames can be better handled, thereby improving the motion compensation result. However, the existing video compression methods based on deep learning are relatively complex. These high-complexity end-to-end video compression method network models can only run on GPUs. The millions of parameters make the models impossible to deploy on side embedded devices, which seriously limits their application and promotion.

[0004] Existing traditional video compression algorithms use manually designed modules and fixed algorithms, resulting in low compression efficiency and poor adaptability. End-to-end video compression algorithms based on deep learning are difficult to deploy on side embedded devices due to the large number of model parameters and computational complexity, and their application is limited. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a video compression system and method based on an embedded platform to solve the problems of low compression efficiency, poor adaptability and large computational complexity in the prior art, which leads to limited application. The present invention aims to design a high-compression-efficiency, lightweight end-to-end video compression model and deploy it on a domestic embedded development platform, thereby solving the above-mentioned problems and further promoting the application of deep learning-based video compression models on side-end embedded devices.

[0006] The basic idea of ​​implementing the present invention is: by building an end-to-end video compression model and training it, the trained video compression model is deployed to the embedded platforms of the compression end and the decompression end respectively. In order to improve the compression efficiency on the embedded platform, the method of skipping entropy coding is used to reduce the time of entropy coding and the residual block pruning is used to lightweight the model. The present invention enables the deep learning-based video compression model to be deployed on the domestic embedded development platform, solves the problem that the existing deep compression model is limited in application due to the high model complexity, large number of parameters and large amount of calculation, and promotes the application of the deep learning-based video compression model on the side embedded devices.

[0007] The specific technical scheme of the present invention to achieve the above purpose is as follows:

[0008] A video compression system based on an embedded platform includes a video acquisition module, a video compression module, a data storage module, a data transmission module and a video decompression module; wherein,

[0009] The video acquisition module is used to collect original video information and upload it to the video compression module;

[0010] The video compression module is composed of a feature extraction submodule, a motion estimation submodule, a motion compression submodule, a first motion decompression submodule, a first motion compensation submodule, a residual compression submodule and a first residual decompression submodule; the module is deployed on the domestic NPU embedded device at the compression end, and is used to compress the original video uploaded by the video acquisition module, obtain a motion vector binary file and a residual binary file, and output the file to the data storage module;

[0011] The data storage module is used to store the data generated by the video compression module;

[0012] The data transmission module is used to send the data in the data storage module to the video decompression module over the communication link;

[0013] The video decompression module is deployed on the domestic NPU embedded device at the decompression end, and consists of a second motion decompression submodule, a second motion compensation submodule, a second residual decompression submodule and a feature reconstruction submodule, and is used to decode the input binary file and obtain the reconstructed video as the final output.

[0014] Furthermore, the above-mentioned video acquisition module uses any hardware device that can acquire images, including mobile phones, professional cameras, monitoring equipment and aerial photography drones; the data transmission module uses physical transmission equipment, including coaxial cables and optical fibers; the input and output of the data storage module and the data transmission module are both binary files.

[0015] Furthermore, the above-mentioned video compression module compresses the original video uploaded by the video acquisition module, specifically: first, the feature extraction submodule is used to extract features of the current frame and the reference frame; then, the motion estimation submodule is used to perform motion estimation on the features of the current frame and the reference frame in the feature domain to obtain a motion vector, and then the motion compression submodule encodes the vector to obtain a motion vector binary file, and uses the first motion decompression submodule to decode the file to obtain a first reconstructed motion vector; the first reconstructed motion vector and the reference frame are sent to the first motion compensation submodule for motion compensation to obtain a first predicted frame feature; finally, the residual of the current frame and the predicted frame features is sent to the residual compression submodule to obtain a residual binary file, and the file is sent to the first residual decompression submodule to obtain a first reconstructed residual feature, and the reference frame feature is updated using the sum of the reconstructed residual feature and the predicted frame feature for compression of the next frame.

[0016] Furthermore, the above-mentioned video decompression module decodes the input binary file, specifically: first, the input motion vector binary file is decoded into a second reconstructed motion vector through the second motion decompression submodule, and then the second reconstructed motion vector and the reference frame feature are sent to the second motion compensation submodule for motion compensation to obtain a second predicted frame feature; then the residual binary file is sent to the second residual decompression submodule to obtain a second reconstructed residual feature; finally, the reconstructed residual feature is added to the predicted frame feature as the reconstructed frame feature, and at the same time updated as the reference frame feature for the next frame decompression; and the feature reconstruction submodule is used to reconstruct the reconstructed frame feature to obtain a reconstructed frame.

[0017] A video compression method based on an embedded platform comprises the following steps:

[0018] (1) Use public video data to construct a sample set and divide it into a training set and a test set; and pre-process the training set to set the frame interval GOP;

[0019] (2) Build an end-to-end video compression model based on deep learning. The specific steps are as follows:

[0020] (2.1) A feature extraction module is constructed using a convolutional layer and a residual layer to encode the input frame and the reference frame into a feature space and obtain feature representations of the input frame and the reference frame;

[0021] (2.2) Construct a motion estimation module, in which the features of the input frame and the reference frame are first concatenated along the channel dimension, and then processed through two convolutional layers and activation functions to output the feature offset of motion estimation;

[0022] (2.3) constructing a motion compression / decompression module, taking the feature offset of motion estimation as the input of the module, extracting features of the feature offset through motion transformation in the module, generating offset features, quantizing the offset features and performing entropy coding; obtaining a compressed bitstream through entropy coding, entropy decoding the bitstream to obtain decoded offset features, and reconstructing the feature offset through a motion inverse transformation network; wherein the motion transformation network is composed of three convolutional layers and two residual layers alternatingly; the entropy coding is an improved entropy coding network with an additional skip entropy coding prediction module, wherein the skip entropy coding prediction module is composed of two convolutional layers, a Leaky ReLU activation function layer and a Gumbel Softmax / Max, and maps the input prior distribution parameters into a binary 0 / 1 mask for skipping the coding step;

[0023] (2.4) Using the reference frame features and motion offset as inputs to the deformable convolution layer, the deformable convolution operation is used to perform motion compensation to obtain the distorted features, and then the distorted features are refined through two convolution layers to produce the final predicted frame;

[0024] (2.5) The network structure of residual compression is the same as that of motion compression and decompression, and both include residual transformation, residual inverse transformation, entropy coding and entropy decoding modules; they are used to compress and decompress the residual of the predicted frame features and the current frame features to obtain the reconstructed residual features;

[0025] (2.6) After adding the predicted frame to the reconstructed residual feature, the predicted frame is processed by a feature reconstruction module to generate a reconstructed output frame; the feature reconstruction module has the same network structure as the feature extraction module, that is, it is composed of one convolutional layer and one residual layer;

[0026] (2.7) An end-to-end video compression model based on deep learning is formed by a feature extraction module, a motion estimation module, a motion compression / decompression module, a motion compensation module, a residual compression / decompression module and a feature reconstruction module;

[0027] (3) using the training set to train the video compression model constructed in step (2) until the model parameters converge, saving the model weight parameters and model structure that perform best on the test set, and obtaining the trained video compression model;

[0028] (4) Fine-tune the trained model and perform lightweight processing through residual block pruning:

[0029] (4.1) Scale the output of the residual block using a scaling factor;

[0030] (4.2) Add sparse regularization to the scaling factor and optimize it by stochastic accelerated proximal gradient APG;

[0031] Introduce a scaling factor β for each residual block in the network, and then add a sparse constraint to β during training to obtain a sparse β;

[0032] (4.3) Delete the residual block structure with β=0 in all residual blocks to complete the lightweight processing;

[0033] (5) Perform equivalent conversion on the video compression model into a model format supported by the AI ​​hardware platform during the deployment phase;

[0034] (6) Decompose the equivalently converted model into a compression model and a decompression model, and use a custom symbolic function to map the deformable convolution operations in the two parts of the model to the corresponding onnx operator operations, save the model structure and parameters after the equivalent conversion as an onnx operator reasoning architecture model file, and then use the model conversion tool provided by the computing platform to convert the onnx operator reasoning architecture model file into a deployment model file OM; deploy the OMs of the two parts of the model to the embedded platforms at both ends respectively;

[0035] (7) Capture video and extract video frames frame by frame for preprocessing. Input the preprocessed video into the encoding end of the trained video compression model, compress it into a bitstream and pass it to the storage module for storage. Finally, the bitstream is transferred from the storage module to the decoding end through the data transmission module for decompression to obtain the reconstructed video frame.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] First, since the present invention takes into account the large number of residual blocks in the compression network model, the residual blocks are scaled and pruned during the training process to build a more lightweight and compact network model; that is, a parameter called a scaling factor is used to scale the output of the residual block, sparse regularization is added to these factors, and the optimization problem is solved by an improved random accelerated proximal gradient APG method; by forcing certain factors to be zero, the corresponding structure is safely deleted, and the unimportant parts of the network are pruned, effectively reducing the number of network parameters and the amount of calculation.

[0038] Second, the present invention takes into account that most elements in the feature map have been accurately predicted by the super-prior network, and thus uses a skip entropy coding method in the motion and residual entropy coding part, skipping the entropy coding of most accurately predicted features, and only entropy coding a few inaccurately predicted features, thereby shortening the time for entropy coding and entropy decoding; it can significantly improve the efficiency of entropy coding and shorten the time for entropy coding and entropy decoding.

[0039] Third, the present invention solves the problem that deep learning-based video compression algorithms are difficult to deploy. It accelerates the speed of video compression on the neural network processing unit NPU by optimizing entropy coding and lightweight models. At the same time, by exporting the model using a custom deformable convolution operator, it can be deployed and run on the Huawei Ascend Atlas 200IDK A2 embedded platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The overall framework diagram of the video compression system constructed in the present invention;

[0041] Figure 2 Flow chart for realizing the method of the present invention;

[0042] Figure 3 It is a working flow chart of the video compression system in the present invention;

[0043] Figure 4 It is a schematic diagram of the structure of the video compression algorithm used in the present invention;

[0044] Figure 5 Schematic diagram of the network structure of the motion and residual compression module in the present invention;

[0045] Figure 6 It is a schematic diagram of the structure of the skip entropy coding module in the present invention;

[0046] Figure 7 A schematic diagram of the structure of the motion compensation module in the present invention

[0047] Figure 8 Schematic diagram of the residual block pruning process in the present invention;

[0048] Fig. 9 This is a flowchart for implementing the custom operator in the present invention;

[0049] Fig.10 A comparison chart of simulation test results provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described clearly and completely below in conjunction with the accompanying drawings.

[0051] Embodiment 1: Refer to the attached Figure 1The present invention proposes a video compression system based on an embedded platform, comprising a video acquisition module, a video compression module, a data storage module, a data transmission module and a video decompression module; wherein the video acquisition module adopts any hardware device capable of acquiring images, such as a mobile phone, a professional camera, a monitoring device and an aerial photography drone, etc., to acquire original video information and upload it to the video compression module; the video compression module is composed of a feature extraction submodule, a motion estimation submodule, a motion compression submodule, a first motion decompression submodule, a first motion compensation submodule, a residual compression submodule and a first residual decompression submodule; the module is deployed on a domestic NPU embedded device at the compression end, and is used to compress the original video uploaded by the video acquisition module, Obtain a motion vector binary file and a residual binary file, and output the file to a data storage module; the data storage module is used to store data generated by a video compression module; the data transmission module uses a physical transmission device, such as a coaxial cable, an optical fiber, etc., to send the data in the data storage module to the video decompression module on a communication link; in this embodiment, the input and output of the above-mentioned data storage module and data transmission module are both binary files; the video decompression module is deployed on a domestic NPU embedded device at the decompression end, and is composed of a second motion decompression submodule, a second motion compensation submodule, a second residual decompression submodule and a feature reconstruction submodule, and is used to decode the input binary file and obtain the reconstructed video as the final output.

[0052] The above-mentioned video compression module compresses the original video uploaded by the video acquisition module, specifically: first, the feature extraction submodule is used to extract features of the current frame and the reference frame; then, the motion estimation submodule is used to perform motion estimation on the features of the current frame and the reference frame in the feature domain to obtain a motion vector, and then the motion compression submodule encodes the vector to obtain a motion vector binary file, and uses the first motion decompression submodule to decode the file to obtain a first reconstructed motion vector; the first reconstructed motion vector and the reference frame are sent to the first motion compensation submodule for motion compensation to obtain a first predicted frame feature; finally, the residual of the current frame and the first predicted frame features is sent to the residual compression submodule to obtain a residual binary file, and the file is sent to the first residual decompression submodule to obtain a first reconstructed residual feature, and the reference frame feature is updated using the sum of the first reconstructed residual feature and the predicted frame feature for compression of the next frame.

[0053] The above-mentioned video decompression module decodes the input binary file, specifically: first, the input motion vector binary file is decoded into a second reconstructed motion vector through the second motion decompression submodule, and then the second reconstructed motion vector and the reference frame feature are sent to the second motion compensation submodule for motion compensation to obtain the second predicted frame feature; then the residual binary file is sent to the second residual decompression submodule to obtain the second reconstructed residual feature; finally, the second reconstructed residual feature is added to the predicted frame feature as the reconstructed frame feature, and at the same time updated as the reference frame feature for the next frame decompression; the feature reconstruction submodule is used to reconstruct the reconstructed frame feature to obtain the reconstructed frame.

[0054] Embodiment 2: Refer to the attached Figure 2 The present invention proposes a video compression method based on an embedded platform, which specifically includes the following steps:

[0055] Step 1. Use public video data to construct a sample set, and divide it into a training set and a test set; and pre-process the training set to set the frame interval GOP. The pre-processing process in this embodiment includes: for the video data in the training set, single-frame training is performed in the early stage of training, that is, two frames of pictures are used as a group to compress the video frame once each time; multi-frame training is performed in the later stage of training, that is, N pictures are used as a group, and the first picture of each group is used as a key frame, that is, a starting reference frame, and video frame compression is performed continuously N-1 times. During this period, each reconstructed frame is used as a reference frame for the next frame compression.

[0056] Step 2. Build an end-to-end video compression model based on deep learning. The specific steps are as follows:

[0057] (2.1) A feature extraction module is constructed using a convolutional layer and a residual layer to encode the input frame and the reference frame into a feature space and obtain feature representations of the input frame and the reference frame;

[0058] (2.2) Construct a motion estimation module, in which the features of the input frame and the reference frame are first concatenated along the channel dimension, and then processed through two convolutional layers and activation functions to output the feature offset of motion estimation;

[0059] (2.3) Constructing a motion compression / decompression module, taking the feature offset of motion estimation as the input of the module, extracting features of the feature offset through the transformation network in the module, generating offset features, quantizing the offset features and performing entropy coding; obtaining a compressed bitstream through entropy coding, entropy decoding the bitstream to obtain decoded offset features, and obtaining reconstructed feature offsets through an inverse transformation network; wherein the motion transformation network is composed of three convolutional layers and two residual layers alternately; the inverse transformation network is composed of three deconvolutional layers and two residual layers alternately; the entropy coding is an improved entropy coding network with an additional skip entropy coding prediction module, wherein the skip entropy coding prediction module is composed of two convolutional layers, a Leaky ReLU activation function layer and a Gumbel Softmax / Max, and maps the input prior distribution parameters into a binary 0 / 1 mask for skipping the coding step;

[0060] The improved entropy coding network only needs to encode the prior information of the corresponding position where the mask information is 1 during entropy coding, and the prior information of the corresponding position where the mask information is 0 is not encoded, that is, the encoding is skipped; during entropy decoding, the pixels of the prior information of the corresponding position where the mask is 0 also do not need to be entropy decoded, and the mean value is directly used to replace the value of the corresponding pixel point, and only the prior information of the corresponding position where the mask information is 1 needs to be decoded.

[0061] Here, regarding the entropy coding optimization method, C2F (Coarse-to-Fine), a coarse-to-fine method, is also considered. A super-prior guided mode prediction network is proposed to predict the skip mode of effective residual coding, and the skipped elements are set to zero. The purpose of fast entropy coding achieved by adding a skip entropy coding prediction module in this embodiment can also be achieved.

[0062] (2.4) Using the reference frame features and motion offset as inputs to the deformable convolution layer, the deformable convolution operation is used to perform motion compensation to obtain the distorted features, and then the distorted features are refined through two convolution layers to produce the final predicted frame;

[0063] (2.5) The network structure of the residual compression / decompression module is the same as that of the motion compression / decompression module, both of which include transformation, inverse transformation, quantization, entropy coding and entropy decoding; and are used to compress and decompress the residual of the predicted frame features and the current frame features to obtain the reconstructed residual features;

[0064] In this embodiment, the residual information is firstly extracted through the transformation network to generate residual features; then the residual features are entropy encoded, and the improved entropy coding network is also used in the entropy coding process to accelerate the entropy coding process; the bit stream obtained by entropy coding is entropy decoded to obtain decoded residual features, which are sent to the residual inverse transformation network to obtain reconstructed residual features.

[0065] (2.6) After adding the predicted frame to the reconstructed residual feature, the predicted frame is processed by a feature reconstruction module to generate a reconstructed output frame; the feature reconstruction module has the same network structure as the feature extraction module, that is, it is composed of one convolutional layer and one residual layer;

[0066] (2.7) An end-to-end video compression model based on deep learning is formed by a feature extraction module, a motion estimation module, a motion compression / decompression module, a motion compensation module, a residual compression / decompression module and a feature reconstruction module;

[0067] Step 3. Use the training set to train the video compression model constructed in step 2 until the model parameters converge, save the model weight parameters and model structure that perform best on the test set, and obtain the trained video compression model. In this embodiment, the training process uses the following loss function:

[0068]

[0069] Among them, R o and R r Respectively represent the number of bits for encoding the motion offset and residual, Represents the input frame X t and reconstruct the frame , and λ is a hyperparameter used to control the rate-distortion trade-off.

[0070] Step 4. Fine-tune the trained model and perform lightweight processing through residual block pruning. The implementation steps are as follows:

[0071] (4.1) Introduce a scaling factor β for each residual block in the network, add a sparse constraint to β during training, and optimize it by stochastically accelerating the proximal gradient to obtain a sparse β;

[0072] (4.2) When β is less than the threshold, it is set to 0, and all residual blocks with β of 0 are deleted to complete the lightweight processing;

[0073] The residual block pruning is optimized according to the following formula during training:

[0074] r i+1 =r i +β i F i (r i ,W i ),

[0075] Among them, r i and r i+1 is the input and output of the ith residual block, F i is the residual function, W iis the weight parameter of the i-th residual block; β i is the scaling factor corresponding to the i-th residual block. After adding sparse constraints to β during training, sparse β is obtained. i When it is less than the threshold, it is set to 0, and the ith residual block can be safely deleted because its output does not contribute to subsequent calculations, and this residual block is skipped during the reasoning process. In this embodiment, the lightweight processing can also be implemented using a channel pruning method or other block pruning methods.

[0076] Step 5. Perform equivalent conversion on the video compression model and convert it into a model format supported by the AI ​​hardware platform in the deployment phase;

[0077] Step 6. Decompose the equivalently converted model into two parts: a compressed model and a decompressed model. onnx provides an open source format for AI models of deep learning and traditional ML. It defines an extensible computational graph model, as well as definitions of built-in operators and standard data types. Use custom symbolic functions to map the deformable convolution operations in the two-part model to corresponding onnx operator operations, save the model structure and parameters after equivalent conversion as an onnx operator reasoning architecture model file, and then use the model conversion tool provided by the computing platform to convert the onnx operator reasoning architecture model file into a deployment model file OM; deploy the OMs of the two-part model to the embedded platforms at both ends respectively;

[0078] Step 7. Capture the video and extract the video frames frame by frame for preprocessing. Input the preprocessed video into the encoding end of the trained video compression model, compress it into a bit stream and pass it to the storage module for storage. Finally, the bit stream is transferred from the storage module to the decoding end through the data transmission module for decompression to obtain the reconstructed video frame.

[0079] Embodiment 3: The overall implementation steps of the video compression method proposed in this embodiment are the same as those of Embodiment 2. Figure 2-10 The implementation process of the present invention is further described in detail by giving a specific example:

[0080] like Figure 2As shown, the method steps include: S1. Prepare public video data sets as training sets and test sets respectively. S2. Preprocess the training set and set the frame interval (GOP, Group of Pictures). S3. Build an end-to-end video compression model based on deep learning. S4. Use the training set to train the end-to-end video compression model, and save the optimal model weight parameters and model structure on the test set. S5. After the model training is completed, fine-tune the model and perform lightweight processing to reduce the computational complexity of the model and improve the inference speed. S6. Perform an equivalent conversion on the video compression model and convert it into a model format supported by the AI ​​hardware platform in the deployment phase. S7. Decompose the converted model and deploy the compression model and decompression model to the Ascend Atlas 200IDK A2 embedded platforms at both ends respectively. S8. Collect the video and extract the video frames frame by frame for preprocessing, input them into the encoding end of the video compression model, compress them into a bit stream, and then pass them to the decoding end for decompression to obtain the reconstructed video frames. The specific implementation steps are as follows:

[0081] Step S1: The dataset uses the Vimeo-90k dataset, which is a video compression dataset with a size of 82GB released by Vimeo. The dataset contains more than 90,000 video clips with a resolution of 448×256.

[0082] Step S2: Preprocess the training set video. Single-frame training is performed in the early stage of training, i.e., two frames are used as a group to compress the video frame once. Multi-frame training is performed starting from the 2 millionth batch, i.e., 12 images are used as a group, and the first image of each group is used as the key frame (starting reference frame) to continuously compress the video frame 11 times, during which the reconstructed frame is used as the reference frame for the next frame compression.

[0083] Step S3: Construct an end-to-end video compression model. The model in the training phase is as follows: Figure 4 As shown, it includes feature extraction, motion estimation, motion compression and decompression, motion compensation, residual compression and decompression modules. The specific steps are as follows:

[0084] S31: Feature extraction

[0085] The feature extraction module uses a convolutional layer and a residual layer to encode the input frame and the reference frame into the feature space.

[0086] S32: Motion Estimation

[0087] After obtaining the feature representations of the input frame and the reference frame, they are input into the motion estimation submodule to obtain the estimated feature offset M tIn the motion estimation module, the two input features are first concatenated along the channel dimension, and then passed through two convolutional layers and activation functions to obtain the feature offset M. t .

[0088] S33: Motion Compression and Motion Decompression

[0089] Offset the feature by M t Feed Figure 5 Compression and decompression are performed in the motion compression model. Figure 5 , in the use of transformation network for M t The offset features are further extracted to generate the transformed feature y. The transformation network consists of three convolutional layers and two residual layers alternating. After y is quantized, it is entropy encoded. Before entropy encoding, y first passes through a super-prior encoder module to further extract and compress the statistical information in y to generate a more compact super-prior feature z. The super-prior feature z is then quantized, entropy encoded, and entropy decoded to obtain the quantized super-prior feature. Through a network module, the quantized hyper-prior feature Mapping is performed to obtain the prior distribution parameters (μ, σ). Using the prior distribution N(μ, σ), the quantized y is entropy encoded to obtain a binary bit stream. Entropy decoding of the binary bit stream is performed to obtain Send it to the inverse transformation network to get Right now Figure 4 In

[0090] Entropy coding can only be processed serially on the CPU and cannot be deployed on the NPU for acceleration. In order to further accelerate the entropy coding process, the present invention designs a skip entropy coding method for faster entropy coding of the quantized y. The structure of skip entropy coding is as follows: Figure 6 As shown, first, a skip coding prediction module is used to generate 0 / 1 mask information for each feature in the quantized y, and the mask is used to mask the feature before the prior information is entropy encoded. 1 in the mask indicates that the predicted prior information is uncertain, and the pixels at the corresponding position need to be entropy encoded; 0 indicates that the prior information has been well predicted, and the pixels at the corresponding position do not need to be entropy encoded, and the mean value can be directly used to replace the value of the corresponding pixel. During entropy encoding, only the prior information at the corresponding position with mask information 1 needs to be encoded, and the prior information at the corresponding position with mask information 0 is not encoded, that is, skipped encoding, which greatly reduces the encoding time. During entropy decoding, the pixels of the prior information at the corresponding position with mask information 0 also do not need to be entropy decoded, and the mean value is directly used to replace the value of the corresponding pixel. Only the prior information at the corresponding position with mask information 1 needs to be decoded, which also reduces the decoding time.

[0091] S34: Motion Compensation

[0092] The present invention uses deformable convolution operation to perform motion compensation to generate prediction features. The network structure is as follows Figure 7 As shown in the figure, during the motion compensation process, the deformable convolution layer takes the reference frame features and motion offsets as input for deformable convolution operations. In this way, the learned dynamic offset kernel can be used to better compress videos with complex non-rigid motion patterns and produce more accurate warped features. In order to generate more accurate prediction features, two convolutional layers are used here to further refine the output features of the deformable convolution layer and finally produce the final prediction frame.

[0093] S35: Residual Compression and Decompression

[0094] The residuals of the predicted frame features and the current frame features are compressed and decompressed to obtain the reconstructed residual features. Figure 5 The network structure of motion compression and decompression is exactly the same, including modules such as transformation, inverse transformation, quantization, entropy coding, and entropy decoding.

[0095] S36: Feature reconstruction

[0096] Finally, the residual information will be reconstructed Add back the predicted frame The reconstructed output frame is generated through the feature reconstruction module. The feature reconstruction module has the same network structure as the feature extraction module, using a convolutional layer and a residual layer.

[0097] Step S4: Training details

[0098] During model training, our loss function is the rate-distortion tradeoff:

[0099]

[0100] Where R o and R r Respectively represent the number of bits for encoding the motion offset and residual, Represents the input frame X t and reconstruct the frame The distortion between , where d() represents the mean square error or MS-SSIM (used to measure the perceptual similarity between two images), and λ is a hyperparameter used to control the rate-distortion trade-off. The hyperparameter λ of the trained model is set to 512, 1024, 2048, and 8192. Different λ corresponds to different bitrates. The larger λ is, the higher the bitrate is. The total number of training batches is 2.3 million. In order to improve the generalization ability of the model, avoid falling into the local optimal solution during training, and quickly converge to the global optimal solution, the learning rate decay strategy is adopted in this experiment. The initial learning rate is set to 5e-5 and starts to decay from the 1.9 millionth training batch.

[0101] Step S5: Model lightweighting

[0102] There are many residual blocks in the video compression model, which will affect the inference speed in actual deployment. When fine-tuning the trained model, the present invention accelerates inference and reduces the number of model parameters through residual block pruning. Specifically, a parameter called a scaling factor is used to scale the output of the residual block. Then sparse regularization is added to these parameters, and this optimization problem is solved by an improved stochastic accelerated proximal gradient (APG) method.

[0103] A scaling factor β is introduced for each residual block in the network to scale the specific structure of the residual block, and then a sparse constraint is added to β during training, that is, L1 regularization (Lasso) is applied. The goal is to obtain a sparse β, as shown in the following formula:

[0104] r i+1 =r i +β i F i (r i ,W i )

[0105] where r i and r i+1 is the input and output of the ith block, F i is the residual function, W i is the parameter of the block. i is the scaling factor corresponding to the i-th residual block.

[0106] When β i When it is less than the threshold, it is set to 0, and its output does not contribute to subsequent calculations, that is, the i-th residual block can be safely deleted, and the residual will be directly skipped during the inference process, such as Figure 8 Medium grey block. Figure 8 The structure of residual pruning is illustrated.

[0107] Step S6: Export the onnx operator reasoning architecture model

[0108] After training is completed, the weights are saved and exported as an onnx operator reasoning architecture model. Since deformable convolution is used in our model, but the latest version of Pytorch does not support the direct conversion of the network with deformable convolution into an onnx operator reasoning architecture model, a custom operator is used to implement the deformable convolution function.

[0109] like Fig. 9As shown, we need to define a custom symbolic function in which we map the deformable convolution operation in PyTorch to the corresponding onnx operator operation. PyTorch will call the symbolic function to generate an onnx operator graph structure that matches the corresponding operation logic. Next, register the custom symbolic function through register_custom_op_symbolic and associate it with the target operator. Then use torch.onnx.export to export the onnx operator inference architecture model containing the custom operator. Finally, we need to verify that the exported model can run correctly to ensure that its functionality is consistent with the original model.

[0110] Due to the specific hardware structure of the Ascend Atlas200I AI computing platform, it cannot directly accelerate the reasoning of the network model. Therefore, a tool is required to convert the model for deployment. First, the equivalent converted model structure and parameters are saved as an onnx reasoning architecture model file, and then the model conversion tool ATC provided by the Ascend Atlas200I AI computing platform is used to convert the onnx reasoning architecture model file into an OM deployment model file; the OM model file is deployed for reasoning applications.

[0111] Step S7: Atlas200I platform deployment

[0112] The split models are deployed to the video compression end and the decompression end respectively. In this embodiment, the hardware platforms of the video compression end and the decompression end are both Huawei Ascend Atlas 200IDK A2 platform. Figure 1 The following figure shows the video compression system we have deployed. The model is deployed to the Ascend Atlas 200IDK A2 embedded platform at both ends. The specific implementation is as follows:

[0113] (6.1) Use a custom symbolic function to map the deformable convolution operation in the model to the corresponding onnx operator operation, and save the equivalently converted model structure and parameters as an ONNX inference architecture model file;

[0114] (6.2) Use the model conversion tool ATC provided by the Ascend Atlas200I AI computing platform to convert the ONNX model file into an OM deployment model file;

[0115] (6.3) Deploy the OM model files of the compression and decompression ends to the Ascend Atlas 200IDK A2 embedded platforms at both ends;

[0116] Step S8: Reasoning Application

[0117] The video acquisition device collects the video and extracts the video frames frame by frame for preprocessing such as image cropping, and then transmits it to the video compression module. After compression, the code stream is transmitted to the storage module for storage. Finally, the code stream is transmitted from the storage module to the video decompression module through the data transmission module for video decompression. The workflow diagram is as follows Figure 3 shown.

[0118] The effect of the present invention is further described below in conjunction with simulation experiments.

[0119] 1. Simulation conditions:

[0120] The simulation experiment of the present invention is carried out in the hardware environment of Huawei Ascend Atlas 200IDK A2 platform and the software environment of Linux operating system.

[0121] 2. Simulation content

[0122] The rate-distortion performance of the video compression model constructed in the present invention was compared with the benchmark models (DVC, FVC, H.265) on the test data set, as well as the encoding and decoding time before and after the model optimization.

[0123] 3. Simulation results

[0124] PSNR and MS-SSIM are tested on the UVG and MCL_JCV datasets respectively. The test results are compared as follows: Fig.10 The compression model of the present invention is lightweight and deployed on an embedded platform based on FVC. Its performance in terms of PSNR and MS-SSIM is slightly lower than that of FVC, but it is still significantly better than traditional H.265 and DVC based on deep learning.

[0125] Using the UVG dataset as the test set, the training hyperparameter λ is 2048, and the video resolution is 1080p, the average compression and decompression time of one frame before and after optimization is shown in the following table:

[0126] Model Model before optimization Skip entropy coding Lightweight Optimized model Total speedup Compression time 1006ms 706ms 878ms 578ms 1.74 Decompression time 698ms 469ms 597ms 368ms 1.89

[0127] The compression time of the compression model of the present invention before optimization is 1006ms, and the decompression time is 698ms; in the ablation experiment, the compression time of the model with only skip entropy coding added is 706ms, and the decompression time is 469ms; the compression time of the model with only lightweight processing is 878ms, and the decompression time is 597ms; the compression time of the model with skip entropy coding and lightweight processing added is 578ms, and the decompression time is 368ms. Compared with the model before optimization, the compression speedup ratio of the present invention is 1.74, and the decompression speedup ratio is 1.89. Under the optimization of the present invention, both compression and decompression time have achieved good speedup ratios.

[0128] The above simulation analysis proves the correctness and effectiveness of the method proposed in the present invention.

[0129] Parts of the present invention that are not described in detail belong to common knowledge among those skilled in the art.

[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Obviously, for professionals in this field, after understanding the content and principles of the present invention, they may make various modifications and changes in form and details without departing from the principles and structures of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A video compression system based on an embedded platform, characterized in that: include: Video acquisition module, video compression module, data storage module, data transmission module and video decompression module; The video acquisition module is used to collect original video information and upload it to the video compression module; The video compression module is composed of a feature extraction submodule, a motion estimation submodule, a motion compression submodule, a first motion decompression submodule, a first motion compensation submodule, a residual compression submodule and a first residual decompression submodule; the module is deployed on the domestic NPU embedded device at the compression end, and is used to compress the original video uploaded by the video acquisition module, obtain a motion vector binary file and a residual binary file, and output the file to the data storage module; The data storage module is used to store the data generated by the video compression module; The data transmission module is used to send the data in the data storage module to the video decompression module over the communication link; The video decompression module is deployed on the domestic NPU embedded device at the decompression end, and consists of a second motion decompression submodule, a second motion compensation submodule, a second residual decompression submodule and a feature reconstruction submodule, and is used to decode the input binary file and obtain the reconstructed video as the final output.

2. The image compression system according to claim 1, characterized in that: The video acquisition module uses any hardware device that can acquire images, including mobile phones, professional cameras, monitoring equipment and aerial photography drones; the data transmission module uses physical transmission equipment, including coaxial cables and optical fibers; the input and output of the data storage module and the data transmission module are both binary files.

3. The image compression system according to claim 1, characterized in that: The video compression module compresses the original video uploaded by the video acquisition module, specifically: first, the feature extraction submodule is used to extract features of the current frame and the reference frame; then, the motion estimation submodule is used to perform motion estimation on the features of the current frame and the reference frame in the feature domain to obtain a motion vector, and then the motion compression submodule encodes the vector to obtain a motion vector binary file, and the first motion decompression submodule is used to decode the file to obtain a first reconstructed motion vector; the first reconstructed motion vector and the reference frame are sent to the first motion compensation submodule for motion compensation to obtain a first predicted frame feature; finally, the residual of the current frame and the first predicted frame features is sent to the residual compression submodule to obtain a residual binary file, and the file is sent to the first residual decompression submodule to obtain a first reconstructed residual feature, and the reference frame feature is updated using the sum of the first reconstructed residual feature and the first predicted frame feature for compression of the next frame.

4. The image compression system according to claim 1, characterized in that: The video decompression module decodes the input binary file, specifically: first, the input motion vector binary file is decoded into a second reconstructed motion vector through the second motion decompression submodule, and then the second reconstructed motion vector and the reference frame feature are sent to the second motion compensation submodule for motion compensation to obtain a second predicted frame feature; then the residual binary file is sent to the second residual decompression submodule to obtain a second reconstructed residual feature; finally, the second reconstructed residual feature is added to the second predicted frame feature as a reconstructed frame feature, and is simultaneously updated as a reference frame feature for next frame decompression; and the feature reconstruction submodule is used to perform feature reconstruction on the reconstructed frame feature to obtain a reconstructed frame.

5. A video compression method based on an embedded platform, characterized in that: The steps include: (1) Use public video data to construct a sample set and divide it into a training set and a test set; and pre-process the training set to set the frame interval GOP; (2) Build an end-to-end video compression model based on deep learning. The specific steps are as follows: (2.1) A feature extraction module is constructed using a convolutional layer and a residual layer to encode the input frame and the reference frame into a feature space and obtain feature representations of the input frame and the reference frame; (2.2) Construct a motion estimation module, in which the features of the input frame and the reference frame are first concatenated along the channel dimension, and then processed through two convolutional layers and activation functions to output the feature offset of motion estimation; (2.3) Construct a motion compression / decompression module, take the feature offset of motion estimation as the input of the module, extract the feature offset through the transformation network in the module, generate the offset feature, quantize the offset feature and perform entropy coding to obtain the compressed bitstream, perform entropy decoding on the bitstream to obtain the decoded offset feature, and then obtain the reconstructed feature offset through the inverse transformation network; The transformation network is composed of three convolutional layers and two residual layers alternately; the inverse transformation network is composed of three deconvolutional layers and two residual layers alternately; the entropy coding is an improved entropy coding network with an additional skip entropy coding prediction module, wherein the skip entropy coding prediction module is composed of two convolutional layers, a Leaky ReLU activation function layer and a GumbelSoftmax / Max, and maps the input prior distribution parameters into a binary 0 / 1 mask for skipping the coding step; (2.4) Using the reference frame features and motion offset as inputs to the deformable convolution layer, the deformable convolution operation is used to perform motion compensation to obtain the distorted features, and then the distorted features are refined through two convolution layers to produce the final predicted frame; (2.5) The network structure of the residual compression / decompression module is the same as that of the motion compression / decompression module, both of which include transformation, inverse transformation, quantization, entropy coding and entropy decoding; and are used to compress and decompress the residual of the predicted frame features and the current frame features to obtain the reconstructed residual features; (2.6) After adding the predicted frame to the reconstructed residual feature, the predicted frame is processed by a feature reconstruction module to generate a reconstructed output frame; the feature reconstruction module has the same network structure as the feature extraction module, that is, it is composed of one convolutional layer and one residual layer; (2.7) An end-to-end video compression model based on deep learning is formed by a feature extraction module, a motion estimation module, a motion compression / decompression module, a motion compensation module, a residual compression / decompression module and a feature reconstruction module; (3) using the training set to train the video compression model constructed in step (2) until the model parameters converge, saving the model weight parameters and model structure that perform best on the test set, and obtaining the trained video compression model; (4) Fine-tune the trained model and perform lightweight processing through residual block pruning: (4.1) Introduce a scaling factor β for each residual block in the network, add a sparse constraint to β during training, and optimize it by stochastically accelerating the proximal gradient to obtain a sparse β; (4.2) When β is less than the threshold, it is set to 0, and all residual blocks with β of 0 are deleted to complete the lightweight processing; (5) Perform equivalent conversion on the video compression model into a model format supported by the AI ​​hardware platform during the deployment phase; (6) Decompose the equivalently converted model into a compression model and a decompression model, and use a custom symbolic function to map the deformable convolution operations in the two parts of the model to the corresponding onnx operator operations, save the model structure and parameters after the equivalent conversion as an onnx operator reasoning architecture model file, and then use the model conversion tool provided by the computing platform to convert the onnx operator reasoning architecture model file into a deployment model file OM; deploy the OMs of the two parts of the model to the embedded platforms at both ends respectively; (7) Capture video and extract video frames frame by frame for preprocessing. Input the preprocessed video into the encoding end of the trained video compression model, compress it into a bitstream and pass it to the storage module for storage. Finally, the bitstream is transferred from the storage module to the decoding end through the data transmission module for decompression to obtain the reconstructed video frame.

6. The method according to claim 5, characterized in that: The preprocessing of setting the frame interval GOP for the training set described in step (1) is specifically to perform single-frame training on the video data in the training set in the early stage of training, that is, to use two frames of pictures as a group to compress the video frame once each time; and to perform multi-frame training in the later stage of training, that is, to use N pictures as a group, and take the first picture of each group as the key frame, i.e. the starting reference frame, and perform N-1 consecutive video frame compressions, during which each reconstructed frame is used as the reference frame for the next frame compression.

7. The method according to claim 5, characterized in that: The training in step (3) is performed using the following loss function: Among them, R o and R r Respectively represent the number of bits for encoding the motion offset and residual, Represents the input frame X t and reconstruct the frame , and λ is a hyperparameter used to control the rate-distortion trade-off.

8. The method according to claim 5, characterized in that: The residual block pruning in step (4) is performed according to the following formula: r i+1 =r i +β i F i (r i ,W i ), Among them, r i and r i+1 is the input and output of the ith residual block, F i is the residual function, W i is the weight parameter of the i-th residual block; β i is the scaling factor corresponding to the i-th residual block.

9. The method according to claim 5, characterized in that: The improved entropy coding network described in step (2.3) only needs to encode the prior information of the corresponding position where the mask information is 1 during entropy coding, and the prior information of the corresponding position where the mask information is 0 is not encoded, that is, the encoding is skipped; during entropy decoding, the pixels of the prior information of the corresponding position where the mask information is 0 also do not need to be entropy decoded, and the mean is directly used to replace the value of the corresponding pixel point, and only the prior information of the corresponding position where the mask information is 1 needs to be decoded.

10. The method according to claim 5, characterized in that: The reconstructed residual features described in step (2.5) are first extracted from the residual information through residual transformation to generate residual features; then the residual features are entropy encoded, and the improved entropy coding network is also used in the entropy coding process to accelerate the entropy coding process; the bit stream obtained by entropy coding is entropy decoded to obtain the decoded residual features, which are sent to the residual inverse transform network to obtain the reconstructed residual features.

Images