A video compression method based on compact representation of human features

Abstract

The application belongs to the field of video coding, and discloses a video compression method based on compact representation of human body features, which comprises the following steps: 1, constructing a human body video compression framework; using human body skeleton features to represent motion, first, the extracted skeleton features are compressed by a human body feature compression algorithm to remove redundancy, and are transmitted to a decoding end together with a reference frame; then, the original skeleton is restored and input into a generation network together with the reconstructed reference frame to generate a predicted frame; finally, the predicted frame is input into a residual optimization module to further improve quality; 2, designing a human body feature compression algorithm; 3, optimizing feature residual coding; first, depth features of the predicted frame and the original frame are extracted by a feature extractor, then, the residual between the depth features is calculated and input into a context encoder to further reduce the bit rate under the optimization of an entropy model. The application greatly improves the compression rate of human body skeleton information, and can generate reconstructed human body videos with good subjective and objective quality at a lower bit stream.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of video coding, and particularly relates to a video compression method based on compact representation of human body features. BACKGROUND

[0002] In recent years, deep learning-based video coding frameworks have shown good prospects for development. Lu et al. proposed the first end-to-end video compression framework DVC, which uses neural networks to replace traditional motion estimation, motion compensation, residual coding and other modules, and optimizes the entire network through a rate-distortion function. Lin et al. used multiple reference frames for motion compensation and motion vector prediction, effectively eliminating temporal redundancy. Yang et al. proposed a learning-based hierarchical quality and recurrent enhancement video compression method, which compresses video frames of different qualities and improves video quality through a recurrent enhancement network. Hu et al. transferred the coding process to the feature domain and used deformable convolution to estimate the motion offset. Their multi-frame feature fusion module extracts multiple feature representations from the current frame and the previous frame, and uses deformable compensation and non-local attention mechanisms to refine the initial reconstructed features, thereby achieving better frame reconstruction. Habibian et al. treated multiple frames as a whole and used 3D convolution for direct compression. Liu et al. used an image encoder and compressed each frame independently, and proposed an entropy model to explore the temporal correlation of latent features. In addition, Li et al. converted residual coding into conditional coding, using temporal context to optimize the encoder and entropy model. However, these methods either use optical flow distortion or directly use latent representations to generate predicted frames, without taking advantage of the characteristics of human body videos for compression, and there is still a lot of redundancy, and it is not possible to achieve low-rate video transmission.

[0003] Due to the powerful generation capability of neural networks, relatively clear images can be generated from very few features, and videos can be mapped to latent representations, and very low-rate compression can be achieved through variational autoencoders (VAEs) and generative adversarial networks (GANs).

[0004] For example, Wang et al. proposed a human video compression method that converts key frames into texture vectors and inputs them into a generative model along with human skeleton information to directly generate reconstructed videos. Meng et al. proposed a single-person surveillance video compression method that inputs key frames and skeleton information into a generative network, and then performs residual calculation in the pixel domain to obtain the final reconstructed frame. However, the human images generated by these methods may be similar to the original video when observed by the human eye, but there are still significant differences in objective indicators, and if residual operations are performed in the pixel domain, large residuals will be generated.

[0005] In summary, although the researchers have proposed a series of encoding compression methods, these methods do not fully consider the characteristics of human video, so it is necessary to study a method that can effectively compress human video according to the characteristics of human video.

[0006] For human video, the human body can be represented by more efficient skeleton features. Although some methods have used human skeletons to compress video, those methods either do not perform residual coding or perform residual coding in the pixel domain, resulting in a large difference in objective quality from the original video, and the residual generated in the pixel domain requires a large amount of code stream transmission, which has no significant advantage compared with other encoding methods. SUMMARY

[0007] The present application aims to provide a video compression method based on compact representation of human features to solve the above technical problems.

[0008] To solve the above technical problems, the specific technical scheme of a video compression method based on compact representation of human features of the present application is as follows:

[0009] A video compression method based on compact representation of human features, comprising the following steps:

[0010] Step 1: Constructing a human video compression framework;

[0011] Using human skeleton features to represent motion, first, the extracted skeleton features are further compressed by a human feature compression algorithm to remove redundancy, and are transmitted to the decoding end together with the reference frame; then restored to the original skeleton, and input into the generation network together with the reconstructed reference frame to generate a predicted frame; finally input into the residual optimization module to further improve the quality;

[0012] Step 2: Designing a human feature compression algorithm;

[0013] The human skeleton is divided into six parts according to the joints to fully utilize the time domain correlation of each joint, then each joint is respectively calculated with the joint of the previous frame to obtain the residual, if the residual is not 0, the residual value, joint index and current frame index value are recorded, then transmitted to the decoding end through arithmetic coding, and the original skeleton is restored through the residual and index;

[0014] Step 3: Feature residual coding optimization;

[0015] First, the feature extractor extracts the depth features of the predicted frame and the original frame. Then, the residual between the depth features is calculated and input into the context encoder. The bit rate is further reduced under the optimization of the entropy model. At the decoding end, the feature residual is recovered by a context decoder. It is added to the features of the predicted frame and then input into the frame generator to generate the final reconstructed frame. The generated reconstructed frame is simultaneously input into the temporal feature extraction module to extract temporal features. The temporal features are input into the context encoder / decoder and the entropy model to optimize the entire model.

[0016] Furthermore, step 1 includes the following specific steps:

[0017] First, extract the first frame I1. Compress the first frame using an image encoder and then transmit it. The remaining frames are denoted as I2, I3...I... t Skeleton information K for each frame is extracted using a pre-trained skeleton feature extractor. t OpenPose is used as the skeleton feature extractor. The extracted skeleton information is further compressed using a human feature compression algorithm. The compressed skeleton information is transmitted using arithmetic encoding. At the decoder, the compressed skeleton information is losslessly restored to the original skeleton information using a skeleton restoration module. Then compress the The skeleton information K1 of the first frame, and the skeleton K of the encoded frame. t The input is fed into the generator network to generate prediction frames. Using PATN as the generator network, the final encoded frame I t Predicted frames and the skeleton K of the encoded frame t The reconstructed frame is obtained by inputting it into the feature residual optimization network.

[0018] Furthermore, the human feature compression algorithm divides the skeleton information extracted by the skeleton extractor into six parts based on human joints: head, body, left arm, right arm, left leg, and right leg.

[0019] Furthermore, step 2 includes the following specific steps:

[0020] First, use a skeleton extractor to extract the skeleton information K of each frame of the video. t Where t is the index of the current frame, and K is the skeleton information. t It contains 18 key points, listed in ascending order: nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear.

[0021]

[0022] wherein represents the horizontal / vertical coordinate of the i-th key point index of the t-th frame of the skeleton information, and the 18 key points are divided into six parts of head, body, left arm, right arm, left leg and right leg according to the human joints, and the key points contained in each part are as follows: head skeleton information body skeleton information left arm skeleton information right arm skeleton information left leg skeleton information right leg skeleton information The complete skeleton information is recovered using a decoding table, and the decoding table is composed as follows:

[0023] Codec = {(index, s i ,e i )|index ∈ [0, 5], s i < e i}

[0024] wherein index represents the joint index, identifying the skeleton part corresponding to the current record, s i represents the starting frame number of the segment of consecutive 0s of the adjacent frame residual, e i represents the ending frame number of the segment of consecutive 0s of the adjacent frame residual, and each record (index, s i , e i ) represents that the skeleton information of the joint index index is equal between the s i -th frame and the e i -th frame.

[0025] Further, the time domain feature acquisition module in step 3 first extracts multi-scale features from the previously decoded features and

[0026]

[0027] Meanwhile, the decoded human skeleton extracts multi-scale skeleton features and

[0028]

[0029] After that, the extracted skeleton features are multiplied by the multi-scale features respectively through a Sigmoid activation function, and pass through a CBAM module to help the network better understand and utilize the spatial information of the input features, so as to better represent the time domain features and improve the network performance.

[0030]

[0031] where ⊙ represents the dot product operation, and σ represents the Sigmoid activation function,

[0032] After that, the extracted skeleton features are multiplied by the multi-scale features respectively through a Sigmoid activation function, and pass through a CBAM module to help the network better understand and utilize the spatial information of the input features, so as to better represent the time domain features and improve the network performance.

[0033]

[0034] In each layer of the hierarchical structure, the spliced features are first refined through a refine module composed of convolutional layers and residual blocks, and then added to to obtain refined features

[0035]

[0036] Then, a ConvLSTM layer is used to capture information in the entire time domain, and the hidden state output by the previous frame is used to update and forget the data of the previous frame. Finally, a residual block (RES) is used to obtain the time domain features and the hidden state

[0037]

[0038] Further, the step 3 includes the following specific steps:

[0039] First, a time domain feature extraction module is used to obtain the temporal context, and the skeleton information of the current frame and the decoded features of the previous frame are input into the time domain feature extraction module (TFE) to obtain the time domain features

[0040]

[0041] After that, the feature extractor Ef(·) extracts the features F of the encoded frame I t and the predicted frame t ori and F t ​​pred The feature extractor Ef(·) is composed of two convolutional layers and a residual block,

[0042] F t ori = Ef(I t )

[0043]

[0044] where two feature extractors Ef(·) share weights, and F t ori and F t pred are subtracted and the time domain features are input into the context encoder Ec(·) to obtain the latent variable y t ,

[0045]

[0046] The probability of the latent representation y t is then estimated by an entropy model, and then compressed and transmitted to the decoding end to generate The latent representation is modeled using a decomposed entropy model with a hyperprior and a Laplace distribution, and the latent representation of the previous frame that has been decoded is input into the entropy model to enrich the input so that it can better estimate the probability of the latent representation y t ,

[0047]

[0048] where Entropy(·) is the entropy model, Ey(·), Dy(·) are compression and decompression operations.

[0049] Further, the specific operations of the compression are as follows:

[0050] First, the mean μ and scale parameter σ of the Laplace distribution of the latent representation y t are calculated by the entropy model, and then the latent representation y t is standardized into the form of the Laplace distribution,

[0051]

[0052] Then, the standardized value z is quantized to obtain the quantized value q,

[0053] q = round(z)

[0054] where round(·) is the quantization operation, and then the quantized value q is compressed into a bit stream by arithmetic coding and transmitted to the decoding end.

[0055] Further, the specific operation of the decompression is as follows:

[0056] The receiving end receives the quantized value Then, the latent representation y is restored by the mean μ and the scale parameter σ of the Laplace distribution t

[0057]

[0058] Finally, the decompressed latent representation y is input into the context decoder Dc(·) to generate the residual feature And the time domain feature And the feature F of the predicted frame are added and input into the frame generator Frame_G(·) to generate the reconstructed frame t pred And the decoded frame feature Will be input into the time domain feature extraction module to generate the time domain feature,

[0059]

[0060] The video compression method based on the compact representation of human body features has the following advantages:

[0061] The application proposes a human video compression coding framework according to the characteristics of human video, uses human skeleton instead of optical flow to represent motion, and transfers the residual coding operation from the pixel domain to the feature domain. By using human skeleton and performing residual coding in the feature domain, the application can generate better reconstructed video quality at a lower bit rate. The application fully utilizes the characteristics of the motion of each part of the human body, proposes a human feature compression algorithm, divides the human joints into six parts for compression, fully eliminates the time domain redundancy of each joint of the human body, and greatly reduces the code rate consumption. The application performs residual optimization in the feature domain and introduces time domain features to optimize the residual module to reduce the code rate generated by the residual.

[0062] In summary, the application greatly improves the compression rate of human skeleton information, and can generate reconstructed human video with good subjective and objective quality at a lower bit stream. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 Fig. 1 is a schematic diagram of the video compression framework based on the compact representation of human body features of the application;

[0064] Figure 2 Fig. 2 is a schematic diagram of the feature residual optimization network structure of the application;

[0065] ​​​Figure 3 A time-domain feature extraction module structure diagram of the present application. DETAILED DESCRIPTION

[0066] In order to better understand the purpose, structure and function of the present application, the present application will be further described in detail below in combination with the drawings.

[0067] Human video mostly has the following characteristics: 1) human body as the only subject of the video foreground, the change area of the video is in the human body part. 2) human body can be represented by a compact skeleton feature, and the skeleton feature consumes lower bit rate than the motion feature. 3) human body features have time-domain correlation, and different joints of the human body have different time-domain features. 4) the human body image generated by the generative model is generally low in objective quality, and the residual operation in the pixel domain will produce a large residual.

[0068] The present application focuses on the above characteristics, fully utilizes the spatio-temporal redundancy of human video sequences and the compact representation of semantic objects, and specifically proposes a human video compression method.

[0069] The present application is a human video compression method based on compact representation of human body features, comprising the following steps:

[0070] Step 1: constructing a human video compression framework;

[0071] Human body as the only subject of human video can be represented by compact skeleton information, and the human video compression framework proposed in the present application uses human skeleton features instead of optical flow to represent motion, which greatly reduces the required bit rate. First, the extracted skeleton features are further compressed by a human feature compression algorithm, and are transmitted to the decoding end together with the reference frame. Then, the original skeleton is restored and input into the generative network together with the reconstructed reference frame to generate a predicted frame. Since the generative network cannot generate video frames well, it also needs to be input into the residual optimization module to further improve the quality.

[0072] As shown in Figure 1 , first, the first frame I1 is extracted, and the first frame is compressed by an image encoder and then transmitted. The remaining frames are denoted as I2, I3...I t , the skeleton information K t of each frame is extracted by a pre-trained skeleton feature extractor. In the present application, openpose is used as the skeleton feature extractor. The extracted skeleton information is further compressed by a human feature compression algorithm, and the compressed skeleton information is transmitted using arithmetic coding. At the decoder end, the compressed skeleton information is restored to the original skeleton information K by a skeleton restoration module. Then the compressed skeleton information K1 of the first frame, skeleton K of the encoded frame t input into the generation network to generate a predicted frame The present application uses PATN as the generation network, which is a pose transfer model capable of generating a relatively clear human image through a human skeleton and a reference human image. Finally, the encoded frame I t , the predicted frame and the skeleton K of the encoded frame t are input into the feature residual optimization network to obtain a reconstructed frame

[0073] Step 2: design a human feature compression algorithm;

[0074] The human skeleton has temporal redundancy, and if it is directly encoded, a large amount of code rate will be consumed. Therefore, the present application designs a human feature compression algorithm to remove the temporal redundancy of the human skeleton feature to further reduce the bit rate. The present application divides the human skeleton into six parts according to the joints to fully utilize the time domain correlation of each joint, and then each joint is respectively calculated with the joint of the previous frame. If the residual is not 0, the residual value, joint index and index value of the current frame are recorded, and then transmitted to the decoding end through arithmetic coding. The residual and index are continuously restored to the original skeleton. This scheme can further reduce the bit rate of the transmitted feature without damaging the skeleton information.

[0075] In order to fully utilize the characteristics of the movement of each part of the human body, the skeleton information extracted by the skeleton extractor is divided into six parts according to the joints of the human body: head, body, left arm, right arm, left leg and right leg. The advantage of such division is that the time domain features of the human video can be fully utilized, and the redundancy can be greatly reduced. For example, when a person only moves the left arm in a period of time, the other parts of the human joints do not move, which produces great redundancy. By eliminating these redundancies, the proposed skeleton optimization method greatly improves the compression rate compared to direct encoding, and it is lossless and can restore the optimized skeleton to the original skeleton at any time.

[0076] Specifically, first, the skeleton information K t of each frame of the video is extracted using a skeleton extractor, wherein t is the index of the current frame, and the skeleton information K t contains 18 key points, and the key point index is arranged from small to large as follows: nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear,

[0077]

[0078] wherein The horizontal / vertical coordinates of the i-th key point index representing the skeleton information of the t-th frame are divided into six parts, i.e., head, body, left arm, right arm, left leg, and right leg, according to the human joints, and the key points contained in each part are as follows: head skeleton information body skeleton information left arm skeleton information right arm skeleton information left leg skeleton information right leg skeleton information

[0079] In the decompression process, in addition to the residual error between adjacent frames, a decoding table is also needed to restore the complete skeleton information, and the decoding table is composed as follows:

[0080] Codec = {(index, s i , e i ) | index ∈ [0, 5], s i < e i}

[0081] where index represents the joint index, s i represents the start frame number of the segment in which the residual error of adjacent frames is continuously all 0, and e i represents the end frame number of the segment in which the residual error of adjacent frames is continuously all 0. Each record (index, s i , e i ) indicates that the skeleton information of the joint index index is equal between the s i -th frame and the e i -th frame.

[0082] Taking the compression and decompression operation of the head skeleton information as an example, the specific steps are as follows:

[0083] First, the head skeleton information Kh1 of the first frame is extracted, and the start frame number s i and the end frame number e i of the decoding table record are recorded as the current frame index, and then the residual error of each adjacent two frames is calculated, and the obtained residual error is

[0084] ΔKh t = Kh t - Kh t-1

[0085] If the value of the current residual error ΔKh t is 0, the end frame number e i of the current decoding table record is updated to the frame index of the current compressed frame. Otherwise, if the residual error ΔKh t is not 0, the residual error ΔKht , and if the start frame number s i of the current decoding table record is smaller than the end frame number e i , then save the current decoding table record (index, s i , e i ) and update the next decoding table record with the decoding table record start frame number s i and end frame number e i as the current frame index t, otherwise only re-assign the decoding table record start frame number s i and end frame number e i as the current frame index t. Repeat this process until the last frame and the second last frame residual ΔKh N is read and recorded, and the last required transmission of the head skeleton information is

[0086] Kh_transfer = {Codec, Kh1, ΔKh t ... ΔKh N}

[0087] where Codec is the decoding table, Kh1 is the head skeleton information of the first frame, and ΔKh t is the residual value which is not zero.

[0088] Kh_transfer is then transmitted to the decoding end through arithmetic coding.

[0089] At the decoding end, first restore the head skeleton information of the first frame Then read the first decoding table record (index, s i , e i ), if the start frame number s i of the current decoding table record is equal to the current frame index value t, then it means that the head skeleton information from the s i frame to the e i frame is equal, i.e.

[0090]

[0091] According to the record, restore the head skeleton information from the s i frame to the e i frame and update the current frame index value to e i .

[0092] Otherwise, read the next residual ΔKh t+1 , restore the head skeleton information of the next frame, and update the index value of the current frame,

[0093]

[0094] If the index value of the current frame t is still not equal to the start frame number si Then repeat the previous step to continue reading the residual and recovering the skeleton information until the index value t of the current frame equals the starting frame number s recorded in the decoding table. i Recover the sth record based on the decoding table. i Frame to e i The frame header skeleton information is retrieved and the current frame index value is updated to e. i .

[0095] Then read the next decoding table record (index, s) i ,e i Continue to determine the starting frame number s recorded in the current decoding table. i Is it equal to the current frame index value t? Repeat the above process until the head skeleton information is fully recovered. The compression and recovery of the skeleton information of the remaining joints are similar. After recovering the skeleton information of the other joints, the skeletons of each joint are rearranged according to the original skeleton coordinates and combined into the final skeleton information. Then the decoded skeleton information is input into the generator network to further generate prediction frames. In this way, the motion state of different joints of the human body can be fully utilized without losing skeleton information, further reducing the bit rate.

[0096] Step 3: Feature residual coding optimization;

[0097] Since the predicted frames generated by the generative network have a good subjective feel but are usually of poor objective quality, directly calculating residuals in the pixel domain would produce a large amount of residuals. Therefore, this invention designs a feature residual optimization network that performs residual encoding in the feature domain instead of the pixel domain, thereby further improving video quality.

[0098] Specifically, the process begins by extracting depth features from the predicted frame and the original frame using a feature extractor. Then, the residual between these depth features is calculated and input into the context encoder, where the bit rate is further reduced through entropy model optimization. At the decoding end, a context decoder reconstructs the feature residual, adds it to the features of the predicted frame, and inputs it into the frame generator to produce the final reconstructed frame. This reconstructed frame is simultaneously input into the temporal feature extraction module to extract temporal features. These temporal features are then fed into the context encoder / decoder and the entropy model to optimize the entire model.

[0099] The proposed time-domain feature acquisition module is as follows: Figure 3 As shown, ⊙ represents the dot product operation, σ represents the Sigmoid function, C represents channel concatenation, and Extract and Kp_extract represent the feature extraction module and skeleton feature extraction module, respectively. First, the three feature extraction modules (extract(·)) consisting of convolutional layers and residual blocks extract features from the previously decoded data. Extracting multi-scale features and

[0100] Meanwhile, the decoded human skeleton Through 3 skeleton feature extraction modules (Kp_extract(·)) composed of convolutional layers, regularization layers, RELU functions and residual blocks, multi-scale skeleton features are extracted and

[0101]

[0102] After that, the extracted skeleton features are multiplied by the multi-scale features respectively through a Sigmoid activation function, and pass through a CBAM module to help the network better understand and utilize the spatial information of the input features, so as to better represent the time domain features and improve the network performance. CBAM (Convolutional Block Attention Module) is an attention mechanism module used to enhance the performance of convolutional neural networks (CNN), which improves the perception ability of the model by introducing channel attention and spatial attention in CNN.

[0103]

[0104] where ⊙ represents the dot product operation, and σ represents the Sigmoid activation function

[0105] After that, the extracted skeleton features are multiplied by the multi-scale features respectively through a Sigmoid activation function, and pass through a CBAM module to help the network better understand and utilize the spatial information of the input features, so as to better represent the time domain features and improve the network performance. CBAM (Convolutional Block Attention Module) is an attention mechanism module used to enhance the performance of convolutional neural networks (CNN), which improves the perception ability of the model by introducing channel attention and spatial attention in CNN.

[0106]

[0107] In each layer of the hierarchical structure, the spliced features are first refined through a refinement module (refine) composed of convolutional layers and residual blocks, and then added to to obtain refined features

[0108]

[0109] Then, a ConvLSTM layer is used to make the model capture information in the entire time domain. ConvLSTM is a time series model that combines convolutional operations with long short-term memory networks (LSTM), which can effectively process spatial data with temporal correlation. Through the hidden state of the output of the last frame The model can update and forget the data of the previous frame by itself. Finally, a residual block (RES) is used to obtain the time domain features ​​and hidden state

[0110]

[0111] As shown in Figure 2 , first use the time domain feature extraction module to obtain the time context, the skeleton information of the current frame and the features of the previous frame decoded are input into the time domain feature extraction module (TFE) to obtain the time domain features

[0112]

[0113] After that, the feature extractor Ef(·) extracts the features F t of the encoded frame I and the predicted frame t ori and F t pred , the feature extractor Ef(·) consists of two convolutional layers and a residual fast,

[0114] F t ori = Ef(I t )

[0115]

[0116] wherein the two feature extractors Ef(·) share weights. Then F t ori and F t pred are subtracted and the time domain features are input into the context encoder Ec(·) to obtain the latent variable y t of the current residual frame,

[0117]

[0118] After that, the probability of the latent representation y t is estimated by the entropy model, and then compressed and transmitted to the decoding end to generate The present application uses the decomposition entropy model of hyper-prior and Laplace distribution to model the latent representation, and in addition, the latent representation of the previous frame which has been decoded is input into the entropy model to enrich the input so as to better estimate the probability of the latent representation y t .

[0119]

[0120] wherein Entropy(·) is an entropy model, Ey(·), Dy(·) are compression, decompression operations, and the specific operations of compression and decompression are as follows,

[0121] Compression: first, the latent representation y is calculated by the entropy model t The mean μ and scale parameter σ of the Laplace distribution of y t is standardized to the form of the Laplace distribution,

[0122]

[0123] After that, the standardized value z is quantized to obtain the quantized value q,

[0124] q = round(z)

[0125] wherein round(·) is a quantization operation, and then the quantized value q is compressed into a bit stream by arithmetic coding and transmitted to the decoding end.

[0126] Decoding: after receiving the quantized value q , the mean μ and scale parameter σ of the Laplace distribution of the latent representation y t are restored to the latent representation

[0127]

[0128] Finally, the decompressed latent representation and the time domain feature are input into the context decoder Dc(·) to generate the residual feature and the feature F t pred of the predicted frame are added and input into the frame generator Frame_G(·) to generate the reconstructed frame and the decoded frame feature will be input into the time domain feature extraction module to generate the time domain feature.

[0129]

[0130] It can be understood that the present application is described through some embodiments, and those skilled in the art know that various changes or equivalent replacements can be made to these features and embodiments without departing from the spirit and scope of the present application. In addition, under the guidance of the present application, these features and embodiments can be modified to adapt to specific conditions and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application are within the scope of the present application.

Claims

1. A video compression method based on compact representation of human features, characterized in that, Includes the following steps: Step 1: Construct a human body video compression framework; Using human skeleton features to characterize motion, the extracted skeleton features are first further compressed using a human feature compression algorithm to reduce redundancy, and then transmitted to the decoding end along with the reference frame. After that, the original skeleton is restored and input into the generator network along with the reconstructed reference frame to generate the prediction frame. Finally, it is input into the residual optimization module to further improve the quality. Step 2: Design a human feature compression algorithm; The human skeleton is divided into six parts according to the joints to make full use of the temporal correlation of each joint. Then, the residual of each joint is calculated with the joint of the previous frame. If the residual is not 0, the residual value, joint index and the index value of the current frame are recorded. Then, it is transmitted to the decoding end through arithmetic encoding. The original skeleton is continuously restored by the residual and index. Step 3: Feature residual coding optimization; First, the feature extractor extracts the depth features of the predicted frame and the original frame. Then, the residual between the depth features is calculated and input into the context encoder. The bit rate is further reduced under the optimization of the entropy model. At the decoding end, the feature residual is recovered by a context decoder. It is added to the features of the predicted frame and then input into the frame generator to generate the final reconstructed frame. The generated reconstructed frame is simultaneously input into the temporal feature extraction module to extract temporal features. The temporal features are input into the context encoder / decoder and the entropy model to optimize the entire model. First, a temporal feature extraction module is used to obtain the temporal context, extracting the skeleton information of the current frame. Features decoded from the previous frame Both are input into the temporal feature extraction (TFE) module to obtain temporal features. Then the feature extractor Ef(·) extracts the encoded frame I. t and predicted frames Feature F t ori and F t pred The feature extractor Ef(·) consists of two convolutional layers and a residual block. F t ori =Ef(I t ) The two feature extractors Ef(·) share weights, and then F... t ori and F t pred Subtraction and Time Domain Features Together, they are input into the context encoder Ec(·) to obtain the latent variable y of the current residual frame. t , Then, the latent representation y is estimated using an entropy model. t The probability is then compressed and transmitted to the decoding end to generate... The latent representation is modeled using a decomposition entropy model based on the super-prior and Laplace distribution, taking the latent representation already decoded in the previous frame as an example. The input is enriched by feeding it into the entropy model to better estimate the latent representation y. t The probability, Entropy(·) represents the entropy model, and Ey(·) and Dy(·) represent compression and decompression operations, respectively. The specific compression operation is as follows: First, the latent representation y is calculated using the entropy model. t The mean μ and scale parameter σ of the Laplace distribution are then used to represent the latent representation y. t Standardize to the form of a Laplace distribution. The standardized value z is then quantized to obtain the quantized value q. q = round(z) The round(·) operation is used for quantization. The quantized value q is then compressed into a bit stream using arithmetic encoding and transmitted to the decoder.

2. The video compression method based on compact representation of human features according to claim 1, characterized in that, Step 1 includes the following specific steps: First, extract the first frame I1. Compress the first frame using an image encoder and then transmit it. The remaining frames are denoted as I2, I3...I... t Skeleton information K for each frame is extracted using a pre-trained skeleton feature extractor. t OpenPose is used as the skeleton feature extractor. The extracted skeleton information is further compressed using a human feature compression algorithm. The compressed skeleton information is transmitted using arithmetic encoding. At the decoder, the compressed skeleton information is losslessly restored to the original skeleton information using a skeleton restoration module. Then compress the The skeleton information K1 of the first frame, and the skeleton K of the encoded frame. t The input is fed into the generator network to generate prediction frames. Using PATN as the generator network, the final encoded frame I t Predicted frames and the skeleton K of the encoded frame t The reconstructed frame is obtained by inputting it into the feature residual optimization network.

3. The video compression method based on compact representation of human features according to claim 1, characterized in that, The human feature compression algorithm divides the skeleton information extracted by the skeleton extractor into six parts according to human joints: head, body, left arm, right arm, left leg, and right leg.

4. The video compression method based on compact representation of human features according to claim 1, characterized in that, Step 2 includes the following specific steps: First, use a skeleton extractor to extract the skeleton information K of each frame of the video. t Where t is the index of the current frame, and K is the skeleton information. t It contains 18 key points, listed in ascending order: nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear. in The horizontal / vertical coordinates of the index of the i-th keypoint representing the skeleton information in frame t are used to divide these 18 keypoints into six parts according to human joints: head, body, left arm, right arm, left leg, and right leg. The keypoints contained in each part are as follows: Head skeleton information Body skeleton information Left arm skeletal information Right arm skeletal information Left leg skeletal information Right leg skeletal information A decoding table is used to recover the complete skeleton information. The decoding table is structured as follows: Codec={(index,s i ,e i )|index∈[0,5],s i <e i } Where 'index' represents the joint index, identifying the skeletal part corresponding to the current record, and 's'... i e represents the starting frame number of a segment where the residuals of adjacent frames are all zero. i This represents the end frame number of a segment where the residuals of adjacent frames are all zero. Each record (index, s) i ,e i This indicates the skeleton information with joint index 'index' starting from the 's'th joint. i Frame to e i The skeleton information is the same across all frames.

5. The video compression method based on compact representation of human features according to claim 1, characterized in that, Step 3, the temporal feature acquisition module, first extracts features from the previously decoded features using three feature extraction modules (extract(·)) consisting of convolutional layers and residual blocks. Extracting multi-scale features and Meanwhile, the decoded human skeleton Multi-scale skeleton features are extracted using a skeleton feature extraction module (Kp_extract(·)) consisting of three convolutional layers, regularization layers, a ReLU function, and residual blocks. and Subsequently, the extracted skeleton features are multiplied by multi-scale features using a Sigmoid activation function, and then passed through a CBAM module to help the network better understand and utilize the spatial information of the input features, thereby better representing temporal features and improving network performance. Where ⊙ represents the dot product operation, and σ represents the Sigmoid activation function. Then, an upsampling module consisting of sub-pixel convolution and residual blocks is used to... Perform upsampling and compare with The splicing is performed along the channel dimension. In each layer of the hierarchical structure, the spliced ​​features First, a refinement module consisting of convolutional layers and residual blocks is used, followed by... Adding them together yields refined features. Then, a ConvLSTM layer is used to enable the model to capture information across the entire temporal domain, using the hidden state output from the previous frame. The model updates and forgets the data from the previous frame on its own. Finally, it uses a residual block (RES) to obtain temporal features. and hidden state 6. The video compression method based on compact representation of human features according to claim 1, characterized in that, The specific decompression operation is as follows: The receiving end receives the quantized value Then, through the latent representation of y t The latent representation of the Laplace distribution with mean μ and scale parameter σ. Finally, the decompressed latent representation and temporal features The input is fed into the context decoder Dc(·) to generate residual features. Features F of the predicted frame t pred The sums are then fed into the frame generator Frame_G(·) to generate a reconstructed frame. and decoded frame features The input will be fed into the temporal feature extraction module to generate temporal features.

Images