Video compression method based on compact representation of human body features

Through a compact representation method based on human characteristics, the human body skeleton features and feature domain residual coding are used to solve the problems of redundancy and high code rate of human videos in the prior art, and high quality video compression effect is achieved at low code rate.

CN119011840AActive Publication Date: 2024-11-22HANGZHOU DIANZI UNIV

Patent Information

Application Number
CN202410948218.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-22
Estimated Expiration
2044-07-16

AI Technical Summary

Technical Problem

The existing video encoding methods fail to fully utilize the characteristics of human video, resulting in large redundancy, difficulty in realizing video transmission at low code rates, and the code stream generated by pixel domain residual operation is consumed relatively large.

Method used

Using a compact representation method based on human characteristics, the human body skeleton features are used to characterize motion, and residual coding is performed in the feature domain. Through the human body feature compression algorithm and feature residual optimization module, the bit rate is reduced and high-quality reconstructed video is generated.

Benefits of technology

It effectively reduces the bit rate consumption of human videos, improves the quality of reconstructed videos, and realizes human video compression with good subjective and objective quality under lower bit streams.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119011840A_ABST
    Figure CN119011840A_ABST
Patent Text Reader

Abstract

The invention belongs to the field of video coding, and discloses a video compression method based on compact representation of human body features, which comprises the following steps of: 1, constructing a human body video compression framework; the motion is represented by using human body skeleton features, firstly, the extracted skeleton features are compressed to be redundant through a human body feature compression algorithm, and the extracted skeleton features and a reference frame are transmitted to a decoding end together; the original skeleton and the reconstructed reference frame are input into the generative network together to generate a prediction frame; and finally, inputting into a residual optimization module to further improve the quality. 2, designing a human body feature compression algorithm; step 3, feature residual error coding optimization; firstly, depth features of a prediction frame and an original frame are extracted through a feature extractor, then a residual error between the depth features is calculated, the residual error is input into a context encoder, and the bit rate is further reduced under optimization of an entropy model. According to the invention, the compression rate of human skeleton information is greatly improved; a reconstructed human body video with good subjective and objective quality can be generated with a low bit stream.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of video coding, and in particular relates to a video compression method based on compact representation of human body features. Background Art

[0002] In recent years, deep learning-based video coding frameworks have shown good development prospects. Lu et al. proposed the first end-to-end video compression framework DVC, which replaced traditional modules such as motion estimation, motion compensation, and residual coding with neural networks, and jointly optimized the entire network through rate-distortion functions. Lin et al. used multiple reference frames for motion compensation and motion vector prediction, effectively eliminating temporal redundancy. Yang et al. proposed a learning-based hierarchical quality and cyclic enhancement video compression method to compress video frames of different qualities and improve video quality through a cyclic enhancement network. Hu et al. transferred the encoding process to the feature domain and used deformable convolution to estimate motion offsets. Their multi-frame feature fusion module extracted multiple feature representations from the current frame and the previous frame, and used deformable compensation and non-local attention mechanisms to refine the initial reconstructed features, thereby achieving better frame reconstruction. Habibian et al. treated multiple frames as a whole and used 3D convolution to directly compress them. Liu et al. used an image encoder and independently compressed each frame, and proposed an entropy model to explore the temporal correlation of latent features. In addition, Li et al. transformed residual coding into conditional coding and optimized the encoder and entropy model using temporal context. However, these methods, whether using optical flow distortion or directly using latent representation to generate predicted frames, do not utilize the characteristics of human body videos for compression, still have great redundancy, and cannot achieve video transmission at low bit rates.

[0003] Due to the powerful generative ability of neural networks, relatively clear images can be generated with very few features, videos can be mapped into potential representations, and compressed with extremely low bit rates through variational autoencoders (VAEs) and generative adversarial networks (GANs).

[0004] For example, Wang et al. proposed a human video compression method, which converts key frames into texture vectors and inputs them into a generative model with human skeleton information to directly generate a reconstructed video. Meng et al. proposed a single-person surveillance video compression method, which inputs key frames and skeleton information into a generative network, and then performs residual calculations in the pixel domain to obtain the final reconstructed frame. However, although the human images generated by these methods are similar to the original videos in human eye observation, they are significantly different from the original videos in objective indicators. If residual operations are performed in the pixel domain, large residuals will be generated.

[0005] In summary, although researchers have proposed a series of coding compression methods, these methods have not fully considered the characteristics of human body videos. Therefore, it is necessary to study a method that can effectively compress human body videos based on the characteristics of human body videos.

[0006] For human videos, the human body can be represented by skeleton features that are more efficient to transmit. Although some methods have used human skeletons to compress videos, those methods either do not perform residual coding or perform residual coding in the pixel domain, resulting in a large difference in objective quality from the original video. In addition, the residual generated in the pixel domain requires a large amount of bitstream transmission, which has no significant advantages over other coding methods. Summary of the invention

[0007] The present invention aims to provide a video compression method based on compact representation of human features to solve the above technical problems.

[0008] In order to solve the above technical problems, a specific technical solution of a video compression method based on compact representation of human features of the present invention is as follows:

[0009] A video compression method based on compact representation of human features comprises the following steps:

[0010] Step 1: Construct a human video compression framework;

[0011] Human skeleton features are used to characterize motion. First, the extracted skeleton features are further compressed using a human feature compression algorithm and transmitted to the decoder together with the reference frame. Then, the skeleton is restored to its original state and input into the generation network together with the reconstructed reference frame to generate a prediction frame. Finally, the prediction frame is input into the residual optimization module to further improve the quality.

[0012] Step 2: Design a human feature compression algorithm;

[0013] The human skeleton is divided into six parts according to the joints to make full use of the temporal correlation of each joint. Then, each joint is calculated with the joint of the previous frame for residuals. If the residual is not 0, the residual value, joint index and index value of the current frame are recorded. Then, the residual value, joint index and index value of the current frame are transmitted to the decoding end through arithmetic coding, and the original skeleton is continuously restored through the residual and index.

[0014] Step 3: Feature residual coding optimization;

[0015] First, the deep features of the predicted frame and the original frame are extracted through the feature extractor, and then the residual between the deep features is calculated and input into the context encoder. The bit rate is further reduced under the optimization of the entropy model. At the decoding end, the feature residual is restored through a context decoder, added to the features of the predicted frame, and input into the frame generator to generate the final reconstructed frame. The generated reconstructed frame is also input into the time domain feature extraction module to extract the time domain features. The time domain features are input into the context codec and the entropy model to optimize the entire model.

[0016] Furthermore, the step 1 includes the following specific steps:

[0017] First, the first frame I1 is extracted, and the first frame is compressed and transmitted through an image encoder. The remaining frames are recorded as I2, I3...I t , extract the skeleton information K of each frame through the pre-trained skeleton feature extractor t , using openpose as the skeleton feature extractor, the extracted skeleton information is further compressed by the human feature compression algorithm, and the compressed skeleton information is transmitted using arithmetic coding. At the decoder end, the compressed skeleton information is losslessly restored to the original skeleton information through the skeleton recovery module After that, the compressed The skeleton information K1 of the first frame and the skeleton K of the coded frame t Input to the generative network to generate the predicted frame Using PATN as the generating network, the encoded frame I t , predicted frame and the skeleton K of the coded frame t Input into the feature residual optimization network to obtain the reconstructed frame

[0018] Furthermore, the human feature compression algorithm divides the skeleton information extracted by the skeleton extractor into six parts according to human joints: head, body, left arm, right arm, left leg and right leg.

[0019] Furthermore, the step 2 includes the following specific steps:

[0020] First, use the skeleton extractor to extract the skeleton information K of each frame of the video t , where t is the index of the current frame, and the skeleton information K t Contains 18 key points, and the key point indexes are arranged from small to large: nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear,

[0021]

[0022] in Represents the horizontal / vertical coordinates of the ith key point index of the skeleton information in the tth frame. These 18 key points are divided into six parts according to the human body joints: head, body, left arm, right arm, left leg and right leg. The key points contained in each part are as follows: Head skeleton information Body skeleton information Left arm skeleton information Right arm skeleton information Left leg skeleton information Right leg skeleton information Use the decoding table to restore the complete skeleton information. The decoding table is composed as follows:

[0023] Codec={(index,s i ,e i )|index∈[0,5],s i <e i}

[0024] Where index represents the joint index, which identifies the skeleton part corresponding to the current record, s i Indicates the starting frame number of the segment where the residuals of adjacent frames are all zeros. i Indicates the end frame number of the segment where the residuals of adjacent frames are all 0. Each record (index, s i ,e i ) indicates that the skeleton information with joint index index is in the sth i Frame to e i The skeleton information is equal between frames.

[0025] Furthermore, the time domain feature acquisition module in step 3 first extracts the previously decoded features through three feature extraction modules (extract(·)) consisting of a convolutional layer and a residual block. Extracting multi-scale features and

[0026]

[0027] At the same time, the decoded human skeleton Multi-scale skeleton features are extracted through three skeleton feature extraction modules (Kp_extract(·)) consisting of convolutional layers, regularization layers, RELU functions and residual blocks. and

[0028]

[0029] Afterwards, the extracted skeleton features are multiplied with the multi-scale features through a Sigmoid activation function and passed through a CBAM module to help the network better understand and utilize the spatial information of the input features, thereby better representing the temporal features and improving network performance:

[0030]

[0031] Where ⊙ represents the dot product operation, σ represents the Sigmoid activation function,

[0032] Then, an upsampling module consisting of sub-pixel convolution and residual blocks is used to Upsample and compare Splicing is done in the channel dimension,

[0033]

[0034] At each level of the hierarchical structure, the concatenated features First, it passes through a refinement module consisting of a convolutional layer and a residual block, and then Add to get the refined features

[0035]

[0036] Then a ConvLSTM layer is used to enable the model to capture information in the entire time domain, using the hidden state output of the previous frame The model updates and forgets the data of the previous frame. Finally, a residual block (RES) is used to obtain the time domain features. and hidden state

[0037]

[0038] Furthermore, the step 3 includes the following specific steps:

[0039] First, we use the temporal feature extraction module to obtain the temporal context and extract the skeleton information of the current frame. and the features decoded from the previous frame Input into the time domain feature extraction module (TFE) to obtain time domain features

[0040]

[0041] Then the feature extractor Ef(·) extracts the coded frame I t and predicted frame Features F t ori and F tpred , the feature extractor Ef(·) consists of two convolutional layers and a residual block,

[0042] F t ori =Ef(I t )

[0043]

[0044] The two feature extractors Ef(·) share weights, and then F t ori and F t pred Subtraction and time domain features Input together into the context encoder Ec(·) to obtain the latent variable y of the current residual frame t ,

[0045]

[0046] Then the entropy model is used to estimate the potential representation y t The probability is then compressed and transmitted to the decoding end to generate The latent representation is modeled using the hyper-prior and Laplace distribution decomposition entropy model, which transforms the latent representation that has been decoded in the previous frame into Input to the entropy model to enrich the input so that it can better estimate the latent representation y t The probability of

[0047]

[0048] Entropy(·) is the entropy model, Ey(·) and Dy(·) are compression and decompression operations.

[0049] Furthermore, the specific operation of the compression is as follows:

[0050] First, the potential representation y is calculated through the entropy model t The mean μ and scale parameter σ of the Laplace distribution of , which will then potentially represent y t Normalized to the form of Laplace distribution,

[0051]

[0052] Then the standardized value z is quantized to obtain the quantized value q.

[0053] q = round(z)

[0054] Among them, round(·) is a quantization operation, and then the quantized value q is compressed into a bit stream through arithmetic coding and transmitted to the decoding end.

[0055] Furthermore, the specific operation of the decompression is as follows:

[0056] The receiving end receives the quantized value Afterwards, through the potential representation y t The mean μ and scale parameter σ of the Laplace distribution restore the latent representation

[0057]

[0058] Finally, the decompressed potential representation and time domain characteristics Input to the context decoder Dc(·) to generate residual features And the feature F of the predicted frame t pred Add and input to the frame generator Frame_G(·) to generate the reconstructed frame and decoded frame features It will be input into the time domain feature extraction module to generate time domain features.

[0059]

[0060] The video compression method based on compact representation of human features of the present invention has the following advantages:

[0061] The present invention proposes a coding framework for human body video compression based on the characteristics of human body videos, uses the human skeleton instead of optical flow to characterize movement, and transfers the residual coding operation from the pixel domain to the feature domain. By using the human skeleton and performing residual coding in the feature domain, the present invention can generate better reconstructed video quality at a lower bit rate. The present invention makes full use of the characteristics of the movement of various parts of the human body and proposes a human body feature compression algorithm, which divides the human body joints into six parts for compression respectively, fully eliminates the time domain redundancy of each joint of the human body, and greatly reduces the bit rate consumption. The present invention performs residual optimization in the feature domain, and introduces time domain features to optimize the residual module to reduce the bit rate generated by the residual.

[0062] In summary, the present invention greatly improves the compression rate of human skeleton information and can generate reconstructed human body videos with good subjective and objective quality at a lower bit rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A schematic diagram of a video compression framework based on compact representation of human features of the present invention;

[0064] Figure 2 This is a schematic diagram of the feature residual optimization network structure of the present invention;

[0065] Figure 3 It is a schematic diagram of the structure of the time domain feature extraction module of the present invention. DETAILED DESCRIPTION

[0066] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a video compression method based on compact representation of human features of the present invention in conjunction with the accompanying drawings.

[0067] Most human videos have the following characteristics: 1) The human body is the only subject in the foreground of the video, and the changing area of ​​the video is in the human body. 2) The human body can be represented by compact skeleton features, which consume lower bit rate than motion features. 3) Human body features have temporal correlation, and different joints of the human body have different temporal features. 4) Human body images generated by generative models are usually of low objective quality, and residual operations in the pixel domain will produce large residuals.

[0068] The present invention focuses on the above characteristics, fully utilizes the temporal and spatial redundancy of human video sequences and the compact representation of semantic targets, and specifically proposes a human video compression method.

[0069] A video compression method based on compact representation of human features of the present invention comprises the following steps:

[0070] Step 1: Construct a human video compression framework;

[0071] As the only subject of human body video, the human body can be represented by compact skeleton information. The human body video compression framework proposed in the present invention uses human skeleton features instead of optical flow to characterize motion, which greatly reduces the bit rate required for transmission. First, the extracted skeleton features are further compressed to remove redundancy through the human feature compression algorithm and transmitted to the decoding end together with the reference frame. After that, it is restored to the original skeleton and input into the generation network together with the reconstructed reference frame to generate a predicted frame. Since the generation network cannot generate video frames well, it also needs to be input into the residual optimization module to further improve the quality.

[0072] like Figure 1 As shown, first extract the first frame I1, and transmit it after being compressed by the image encoder. The remaining frames are denoted as I2, I3...I t , extract the skeleton information K of each frame through the pre-trained skeleton feature extractor t , the present invention uses openpose as a skeleton feature extractor. The extracted skeleton information is further compressed by the human feature compression algorithm, and the compressed skeleton information is transmitted using arithmetic coding. At the decoder end, the compressed skeleton information is losslessly restored to the original skeleton information by the skeleton recovery module After that, the compressed The skeleton information K1 of the first frame and the skeleton K of the coded frame t Input to the generative network to generate the predicted frame The present invention uses PATN as a generation network. PATN is a posture transfer model that can generate a clearer human image through a human skeleton and a reference human image. t , predicted frame and the skeleton K of the coded frame t Input into the feature residual optimization network to obtain the reconstructed frame

[0073] Step 2: Design a human feature compression algorithm;

[0074] The human skeleton has time redundancy, and if it is directly encoded, a large amount of bit rate will be consumed. Therefore, the present invention designs a human feature compression algorithm to remove the time redundancy of human skeleton features to further reduce the bit rate. The present invention divides the human skeleton into six parts according to the joints to make full use of the time domain correlation of each joint, and then each joint is respectively calculated with the joint of the previous frame. If the residual is not 0, the residual value, joint index and index value of the current frame are recorded, and then transmitted to the decoding end through arithmetic coding, and the original skeleton is continuously restored through the residual and index. This scheme can further reduce the bit rate of the transmission feature without damaging the skeleton information.

[0075] In order to make full use of the characteristics of the movement of various parts of the human body, the skeleton information extracted by the skeleton extractor is divided into six parts according to the human joints: head, body, left arm, right arm, left leg and right leg. The advantage of this division is that it can make full use of the temporal characteristics of the human body video and can greatly reduce redundancy. For example, when a person's left arm is moving for a period of time, the other parts of the human body joints do not move, which produces a lot of redundancy. By eliminating these redundancies, the proposed skeleton optimization method has greatly improved the compression rate compared to direct encoding, and it is lossless, and the optimized skeleton can be restored to the original skeleton at any time.

[0076] Specifically, the skeleton information K of each frame of the video is first extracted using the skeleton extractor t , where t is the index of the current frame, and the skeleton information K t Contains 18 key points, and the key point indexes are arranged from small to large: nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear,

[0077]

[0078] in Represents the horizontal / vertical coordinates of the ith key point index of the skeleton information in the tth frame. We divide these 18 key points into six parts according to the human body joints: head, body, left arm, right arm, left leg and right leg. The specific key points of each part are as follows: Head skeleton information Body skeleton information Left arm skeleton information Right arm skeleton information Left leg skeleton information Right leg skeleton information

[0079] In addition to the residuals between adjacent frames, this algorithm also requires a decoding table to restore the complete skeleton information during the decompression process. The composition of the decoding table is as follows:

[0080] Codec={(index,s i ,e i )|index∈[0,5],s i <e i}

[0081] Where index represents the joint index, which identifies the skeleton part corresponding to the current record, s i Indicates the starting frame number of the segment where the residuals of adjacent frames are all zeros. i Indicates the end frame number of the segment where the residuals of adjacent frames are all 0. Each record (index,s i ,e i ) indicates that the skeleton information with joint index index is in the sth i Frame to e i The skeleton information is equal between frames.

[0082] Taking the compression and decompression of head skeleton information as an example, the specific steps are as follows:

[0083] First, extract the head skeleton information Kh1 of the first frame and set the starting frame number s recorded in the decoding table i and the end frame number e i Recorded as the current frame index, and then calculate the residual of the skeleton of each adjacent frame, the residual obtained is,

[0084] ΔKh t =Kh t -Kh t-1

[0085] If the current residual ΔKh t If the value is 0, the end frame number e recorded in the current decoding table is i Update to the frame index of the current compressed frame. Otherwise, if the residual ΔKh t If it is not 0, first save the residual ΔKh of the skeleton information of the current frame and the previous framet , and if the starting frame number s recorded in the current decoding table i Less than the end frame number e i , then save the current decoding table record (index,s i ,e i ), and update the next decoding table record, and set the decoding table record starting frame number s i and the end frame number e i Record as the current frame index t, otherwise only the decoding table records the starting frame number s i and the end frame number e i Re-record the current frame index t. Repeat this process until the residual ΔKh between the last frame and the second-to-last frame is read N And record, the head skeleton information that needs to be transmitted is

[0086] Kh_transfer={Codec,Kh1,ΔKh t ...ΔKh N}

[0087] Codec is the decoding table, Kh1 is the head skeleton information of the first frame, ΔKh t is a non-zero residual value.

[0088] Then Kh_transfer is transmitted to the decoding end through arithmetic coding.

[0089] At the decoding end, first restore the head skeleton information of the first frame Then read the first decoding table record (index,s i ,e i ), if the current decoding table records the starting frame number s i is equal to the current frame index value t, then it means that from the sth i Frame to e i The head skeleton information of the frames is equal, that is,

[0090]

[0091] Restore the sth according to the record i Frame to e i The head skeleton information of the frame and update the current frame index value to e i .

[0092] Otherwise read the next residual ΔKh t+1 , restore the head skeleton information of the next frame and update the index value of the current frame,

[0093]

[0094] If the index value t of the current frame is still not equal to the starting frame number si , then repeat the previous step, continue to read the residual, and restore the skeleton information until the index value t of the current frame is equal to the starting frame number s recorded in the decoding table i , according to the decoding table record to restore the s i Frame to e i The head skeleton information of the frame and update the current frame index value to e i .

[0095] Then read the next decoding table record (index, s i ,e i ), continue to determine the current decoding table record starting frame number s i Is it equal to the current frame index value t? Repeat the above process until the head skeleton information is completely restored. The compression and restoration of the skeleton information of the remaining joints are similar. After restoring the skeleton information of other joints, the skeletons of each joint are rearranged according to the original skeleton coordinates and combined into the final skeleton information. Then the decoded skeleton information is input into the generation network to further generate the predicted frame. In this way, the motion state of different joints of the human body can be fully utilized without losing any skeleton information, further reducing the bit rate.

[0096] Step 3: Feature residual coding optimization;

[0097] Since the predicted frames generated by the generative network have a good subjective feeling, but the objective quality is usually poor, directly calculating the residual in the pixel domain will produce a large amount of residual. Therefore, the present invention designs a feature residual optimization network, which performs residual encoding in the feature domain instead of the pixel domain, further improving the video quality.

[0098] Specifically, the deep features of the predicted frame and the original frame are first extracted through the feature extractor, and then the residual between the deep features is calculated and input into the context encoder to further reduce the bit rate under the optimization of the entropy model. At the decoding end, the feature residual is restored through a context decoder, added to the features of the predicted frame, and input into the frame generator to generate the final reconstructed frame. The generated reconstructed frame is also input into the time domain feature extraction module to extract the time domain features. The time domain features are input into the context codec and the entropy model to optimize the entire model.

[0099] The proposed time domain feature acquisition module is as follows Figure 3 As shown in Figure 1, ⊙ represents the dot product operation, σ represents the Sigmoid function, C represents channel concatenation, Extract and Kp_extract represent the feature extraction module and the skeleton feature extraction module respectively. First, the three feature extraction modules (extract(·)) consisting of convolutional layers and residual blocks are used to extract the previously decoded features. Extracting multi-scale features and

[0100] At the same time, the decoded human skeleton Multi-scale skeleton features are extracted through three skeleton feature extraction modules (Kp_extract(·)) consisting of convolutional layers, regularization layers, RELU functions and residual blocks. and

[0101]

[0102] After that, the extracted skeleton features are multiplied with the multi-scale features through a Sigmoid activation function and passed through a CBAM module to help the network better understand and utilize the spatial information of the input features, so as to better represent the temporal features and improve the network performance. CBAM (Convolutional Block Attention Module) is an attention mechanism module used to enhance the performance of convolutional neural networks (CNNs). It improves the perception ability of the model by introducing channel attention and spatial attention in CNNs.

[0103]

[0104] Where ⊙ represents the dot product operation and σ represents the Sigmoid activation function

[0105] Then, an upsampling module consisting of sub-pixel convolution and residual blocks is used to Upsample and compare Splicing is done in the channel dimension,

[0106]

[0107] At each level of the hierarchical structure, the concatenated features First, it passes through a refinement module consisting of a convolutional layer and a residual block, and then Add to get the refined features

[0108]

[0109] Then, a ConvLSTM layer is used to enable the model to capture information in the entire time domain. ConvLSTM is a temporal model that combines convolution operations with long short-term memory networks (LSTMs) and can effectively process spatial data with temporal correlations. The model can update and forget the data of the previous frame by itself. Finally, a residual block (RES) is used to obtain the temporal features. and hidden state

[0110]

[0111] like Figure 2 As shown, we first use the time domain feature extraction module to obtain the temporal context and extract the skeleton information of the current frame. and the features decoded from the previous frame Input into the time domain feature extraction module (TFE) to obtain time domain features

[0112]

[0113] Then the feature extractor Ef(·) extracts the coded frame I t and predicted frame Features F t ori and F t pred , the feature extractor Ef(·) consists of two convolutional layers and a residual block,

[0114] F t ori =Ef(I t )

[0115]

[0116] The two feature extractors Ef(·) share weights. t ori and F t pred Subtraction and time domain features Input together into the context encoder Ec(·) to obtain the latent variable y of the current residual frame t ,

[0117]

[0118] Then the entropy model is used to estimate the potential representation y t The probability is then compressed and transmitted to the decoding end to generate The present invention uses a hyper-prior and a decomposition entropy model of the Laplace distribution to model the latent representation, and also uses the latent representation that has been decoded in the previous frame Input to the entropy model to enrich the input so that it can better estimate the latent representation y t probability.

[0119]

[0120] Entropy(·) is the entropy model, Ey(·) and Dy(·) are compression and decompression operations. The specific operations of compression and decompression are as follows:

[0121] Compression: First, the potential representation y is calculated through the entropy model t The mean μ and scale parameter σ of the Laplace distribution of , which will then potentially represent y t Normalized to the form of Laplace distribution,

[0122]

[0123] Then the standardized value z is quantized to obtain the quantized value q.

[0124] q = round(z)

[0125] Among them, round(·) is a quantization operation, and then the quantized value q is compressed into a bit stream through arithmetic coding and transmitted to the decoding end.

[0126] Decoding: The receiver receives the quantized value Afterwards, through the potential representation y t The mean μ and scale parameter σ of the Laplace distribution restore the latent representation

[0127]

[0128] Finally, the decompressed potential representation and time domain characteristics Input to the context decoder Dc(·) to generate residual features And the feature F of the predicted frame t pred Add and input to the frame generator Frame_G(·) to generate the reconstructed frame and decoded frame features It will be input into the time domain feature extraction module to generate time domain features.

[0129]

[0130] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A video compression method based on compact representation of human features, characterized in that: The steps include: Step 1: Construct a human video compression framework; Human skeleton features are used to characterize motion. First, the extracted skeleton features are further compressed using a human feature compression algorithm and transmitted to the decoder together with the reference frame. Then, the skeleton is restored to its original state and input into the generation network together with the reconstructed reference frame to generate a prediction frame. Finally, the prediction frame is input into the residual optimization module to further improve the quality. Step 2: Design a human feature compression algorithm; The human skeleton is divided into six parts according to the joints to make full use of the temporal correlation of each joint. Then, each joint is calculated with the joint of the previous frame for residuals. If the residual is not 0, the residual value, joint index and index value of the current frame are recorded. Then, the residual value, joint index and index value of the current frame are transmitted to the decoding end through arithmetic coding, and the original skeleton is continuously restored through the residual and index. Step 3: Feature residual coding optimization; First, the deep features of the predicted frame and the original frame are extracted through the feature extractor, and then the residual between the deep features is calculated and input into the context encoder. The bit rate is further reduced under the optimization of the entropy model. At the decoding end, the feature residual is restored through a context decoder, added to the features of the predicted frame, and input into the frame generator to generate the final reconstructed frame. The generated reconstructed frame is also input into the time domain feature extraction module to extract the time domain features. The time domain features are input into the context codec and the entropy model to optimize the entire model.

2. The video compression method based on compact representation of human features according to claim 1, characterized in that: The step 1 comprises the following specific steps: First, the first frame I1 is extracted, and the first frame is compressed and transmitted through an image encoder. The remaining frames are recorded as I2, I3...I t , extract the skeleton information K of each frame through the pre-trained skeleton feature extractor t , using openpose as the skeleton feature extractor, the extracted skeleton information is further compressed by the human feature compression algorithm, and the compressed skeleton information is transmitted using arithmetic coding. At the decoder end, the compressed skeleton information is losslessly restored to the original skeleton information through the skeleton recovery module After that, the compressed The skeleton information K1 of the first frame and the skeleton K of the coded frame t Input to the generative network to generate the predicted frame Using PATN as the generating network, the encoded frame I t , predicted frame and the skeleton K of the coded frame t Input into the feature residual optimization network to obtain the reconstructed frame 3. The video compression method based on compact representation of human features according to claim 1, characterized in that: The human feature compression algorithm divides the skeleton information extracted by the skeleton extractor into six parts according to human joints: head, body, left arm, right arm, left leg and right leg.

4. The video compression method based on compact representation of human features according to claim 1, characterized in that: The step 2 comprises the following specific steps: First, use the skeleton extractor to extract the skeleton information K of each frame of the video t , where t is the index of the current frame, and the skeleton information K t Contains 18 key points, and the key point indexes are arranged from small to large: nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear, in Represents the horizontal / vertical coordinates of the ith key point index of the skeleton information in the tth frame. These 18 key points are divided into six parts according to the human body joints: head, body, left arm, right arm, left leg and right leg. The key points contained in each part are as follows: Head skeleton information Body skeleton information Left arm skeleton information Right arm skeleton information Left leg skeleton information Right leg skeleton information Use the decoding table to restore the complete skeleton information. The decoding table is composed as follows: Codec={(index,s i ,e i )|index∈[0,5],s i <e i } Where index represents the joint index, which identifies the skeleton part corresponding to the current record, s i Indicates the starting frame number of the segment where the residuals of adjacent frames are all zeros. i Indicates the end frame number of the segment where the residuals of adjacent frames are all 0. Each record (index, s i ,e i ) indicates that the skeleton information with joint index index is in the sth i Frame to e i The skeleton information is equal between frames.

5. The video compression method based on compact representation of human features according to claim 1, characterized in that: Step 3: The temporal feature acquisition module first extracts the previously decoded features through three feature extraction modules (extract(·)) consisting of a convolutional layer and a residual block. Extracting multi-scale features At the same time, the decoded human skeleton Multi-scale skeleton features are extracted through three skeleton feature extraction modules (Kp_extract(·)) consisting of convolutional layers, regularization layers, RELU functions and residual blocks. and Afterwards, the extracted skeleton features are multiplied with the multi-scale features through a Sigmoid activation function and passed through a CBAM module to help the network better understand and utilize the spatial information of the input features, thereby better representing the temporal features and improving network performance: Where ⊙ represents the dot product operation, σ represents the Sigmoid activation function, Then, an upsampling module consisting of sub-pixel convolution and residual blocks is used to Upsample and compare Splicing is done in the channel dimension, At each level of the hierarchical structure, the concatenated features First, it passes through a refinement module consisting of a convolutional layer and a residual block, and then Add to get the refined features Then a ConvLSTM layer is used to enable the model to capture information in the entire time domain, using the hidden state output of the previous frame The model updates and forgets the data of the previous frame. Finally, a residual block (RES) is used to obtain the time domain features. and hidden state 6. The video compression method based on compact representation of human features according to claim 1, characterized in that: The step 3 comprises the following specific steps: First, we use the temporal feature extraction module to obtain the temporal context and extract the skeleton information of the current frame. and the features decoded from the previous frame Input into the time domain feature extraction module (TFE) to obtain time domain features Then the feature extractor Ef(·) extracts the coded frame I t and predicted frame Features and The feature extractor Ef(·) consists of two convolutional layers and a residual block. The two feature extractors Ef(·) share weights and then and Subtraction and time domain features Input together into the context encoder Ec(·) to obtain the latent variable y of the current residual frame t , Then the entropy model is used to estimate the potential representation y t The probability is then compressed and transmitted to the decoding end to generate The latent representation is modeled using the hyper-prior and Laplace distribution decomposition entropy model, which transforms the latent representation that has been decoded in the previous frame into Input to the entropy model to enrich the input so that it can better estimate the latent representation y t The probability of Entropy(·) is the entropy model, Ey(·) and Dy(·) are compression and decompression operations.

7. The video compression method based on compact representation of human features according to claim 6, characterized in that: The specific operation of the compression is as follows: First, the potential representation y is calculated through the entropy model t The mean μ and scale parameter σ of the Laplace distribution of , which will then potentially represent y t Normalized to the form of Laplace distribution, Then the standardized value z is quantized to obtain the quantized value q. q = round(z) Among them, round(·) is a quantization operation, and then the quantized value q is compressed into a bit stream through arithmetic coding and transmitted to the decoding end.

8. The video compression method based on compact representation of human features according to claim 6, characterized in that: The specific operation of the decompression is as follows: The receiving end receives the quantized value Afterwards, through the potential representation y t The mean μ and scale parameter σ of the Laplace distribution restore the latent representation Finally, the decompressed potential representation and time domain characteristics Input to the context decoder Dc(·) to generate residual features And the feature F of the predicted frame t pred Add and input to the frame generator Frame_G(·) to generate the reconstructed frame and decoded frame features It will be input into the time domain feature extraction module to generate time domain features.

Citation Information

Patent Citations

  • Multi-mode lossless compression implementation method for human skeleton in video

    CN111641830A

  • Human body video compression method and system based on generative model

    CN115052147A

  • Pp bag feeding machine

    KR102474193B1

  • Skeleton-based dynamic point cloud estimation system for sequence compression

    US20220189119A1

Cited By

  • Human body video coding, decoding and communication method and model optimization method

    CN120034653A