Ultra-low bit rate video conferencing compression method based on second-order motion information

Abstract

The present invention discloses an ultra-low bit rate video conference compression method based on second-order motion information. The method comprises the following steps: (1) selecting a source frame and a motion frame from a video frame, and extracting the sparse motion features of the motion frame; (2) encoding and compressing the source frame, quantizing the sparse motion features of the motion frame, performing inter-frame residual to obtain first-order motion residual information, and performing entropy coding; encapsulating two code streams into data packets for transmission; (3) simulating packet loss during transmission, judging whether packet loss occurs, and performing prediction if packet loss occurs; (4) if there is no packet loss or the prediction is completed, dividing the data packet into two parts; (5) processing the data to obtain the source frame and its sparse motion features, the sparse motion features of the current frame, and obtaining the occlusion mask and first-order optical flow information of the current frame; (6) repeating steps (1) to (5) twice, starting from the third frame, using the optical flow information of the first two frames to smooth the optical flow of the current frame, and finally redrawing the optical flow distortion result with the occlusion mask.

Description

Technical Field

[0001] The present invention relates to the field of video compression, and in particular to an ultra-low bit rate video conferencing compression method based on second-order motion information. Background Art

[0002] Video conferencing is widely used in education, business, and entertainment, especially during the epidemic. Currently, commercial systems mainly use well-known standards such as H.264 / AVC, HEVC (High Efficiency Video Coding), and AV1 for service provision, but their quality of experience is often affected under unreliable network connections (such as congestion, channel fading, etc.). In the natural environment, this is because traditional video compression methods based on dense pixel blocks cannot compactly represent the temporal frame dynamics of high-fidelity reconstruction under a very limited bitrate budget, resulting in a loss of experience quality.

[0003] Recently, the deep image animation task has shown potential to address this problem by compressing a small number of source images and very sparse motion features from motion frames for high-quality, full-resolution video generation at the receiver. In this class of tasks, these motion frames can come from another video sequence for the general motion transfer task. However, for simplicity, in conferencing applications, these driving frames are inter-frame (or P) temporal dynamic representations; in contrast, the source images are usually intra-frame (or I) frames from the same video.

[0004] Currently popular deep image animation methods mainly rely on first-order motion models or their variants to build generative models. Although they can produce good quality frames, ignoring second-order motion information will lead to severe temporal jitter in the synthesized videos. In addition, all existing solutions do not take into account network packet loss, which often occurs in real-world conferencing systems. Summary of the invention

[0005] The purpose of the present invention is to propose an ultra-low bit rate video conferencing compression method based on the temporal consistency of second-order motion information.

[0006] The technical solution adopted by the present invention is:

[0007] The ultra-low bit rate video conferencing compression method based on second-order motion information includes the following steps:

[0008] (1) Select a few frames from the video frames and transmit them directly in the form of their own images as source frames, and use the remaining frames as motion frames to extract sparse motion features from the motion frame images;

[0009] (2) Encoding and compressing the source frame; quantizing the sparse motion features and then performing inter-frame residuals, and encoding the first-order motion residual information obtained through entropy coding; encapsulating the two code streams generated by the source frame and the motion frame into a data packet for transmission;

[0010] (3) During the transmission process, simulate the packet loss phenomenon in real scenarios and determine whether packet loss occurs. If packet loss occurs, predict the loss information;

[0011] (4) After determining that no packet loss has occurred or the prediction of packet loss information is completed, the received data packet is divided into two parts: a source frame and first-order motion residual information. Then, sparse motion features are extracted from the source frame, and the first-order motion residual information is accumulated between frames to restore the sparse motion features of the current motion frame.

[0012] (5) obtaining an occlusion mask of the current motion frame and first-order optical flow information from the source frame to the current frame based on the source frame and its sparse motion features obtained in step (4) and the sparse motion features of the current motion frame;

[0013] (6) Repeat steps (1) to (5) twice. Since quantizing sparse motion features will cause errors in the first-order optical flow information, starting from the third frame, the first-order optical flow information obtained in the first two frames is used to smooth the optical flow of the current frame through the second-order optical flow information, and then the occlusion mask of the current frame is used to assist in redrawing the optical flow distortion result to reconstruct the current motion frame.

[0014] Compared with the prior art, the method of the present invention is based on the temporal consistency of the second-order motion information. On the basis of the first-order optical flow used in the reconstructed frame, the second-order optical flow information is introduced and multi-frame weighting is performed. For actual packet loss scenarios, the packet loss is effectively compensated by the second-order motion residual within a two-frame time window, providing a smoother temporal quality in the reconstruction result and being closer to the real video. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the implementation principle of the method of the present invention.

[0016] Figure 2 It is a structural diagram of the second-order packet loss prediction module of the present invention.

[0017] Figure 3 It is a structural diagram of the prediction network in the second-order packet loss prediction module of the present invention.

[0018] Figure 4 It is a schematic diagram of the weighted sum of the second-order optical flow of the present invention. DETAILED DESCRIPTION

[0019] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] The implementation principle diagram of the method of the present invention is as follows Figure 1 The specific steps include:

[0021] First, select a few frames from the video frame to transmit the entire image as the source frame, and the remaining frames are used as motion frames to extract only the motion information in the image. The source frame is encoded and compressed using existing video compression standards such as H.264 and H.265; sparse motion features are extracted from the motion frame (this embodiment selects 10 key points and a 2x2 affine matrix for each key point), the sparse motion features are first quantized from float32 to int8 and then inter-frame residuals are made, and finally the processed first-order motion residual information is encoded by entropy coding (such as arithmetic coding). The two code streams of the source frame and the inter-frame first-order motion residual information of the motion frame are encapsulated into a data packet for transmission. The received data packet is divided into two parts: the source frame and the first-order motion residual information. The source frame is decoded by the corresponding decoder, and the inter-frame first-order motion residual information of the motion frame is decoded by entropy decoding (such as arithmetic decoding) and it is determined whether packet loss occurs during transmission. If there is no packet loss, inter-frame accumulation is directly performed to recover the sparse motion features of each current motion frame; if packet loss occurs, it is assumed that only one packet is lost each time, that is, the motion residual information of a frame time window size is lost, then the packet loss information is predicted by the packet loss prediction module and then inter-frame accumulation is performed to recover the sparse motion features of each current motion frame.

[0022] In order to reconstruct each current frame, for the first two frames, the sparse motion information of the current frame and the sparse motion information of the source frame and the source frame are sent to the dense motion network, which is mainly composed of the hourglass network. The output of the hourglass network is passed through a convolutional layer to obtain the occlusion mask for generating the current frame. The output of the hourglass network is passed through a convolutional layer and an activation function to obtain the first-order optical flow information from the source frame to the current frame. The source image is distorted only by the first-order optical flow information, and then the occlusion mask is used to redraw the parts of the distorted image where the optical flow cannot be distorted well due to motion occlusion, thereby reconstructing the current frame. From the third frame onwards, the optical flow smoothing method is used to obtain the weighted and smoothed optical flow using the second-order optical flow information obtained by using multiple frames of optical flow. The source image is distorted with this weighted sum of optical flow information, and then the occlusion mask is used to redraw the parts of the distorted image where the optical flow cannot be distorted well due to motion occlusion, thereby reconstructing the current frame.

[0023] The structure diagram of the packet loss prediction module of the present invention is as follows Figure 2 , Figure 3Here, it is assumed that when the packet at time t+1 is lost, the first-order motion residual information contained in the three packets received at time t, t-1, and t-2 are subtracted from each other to obtain the second-order motion residual information at time t-1 and t-2, and sent to the prediction network to predict the second-order motion residual information at time t and add it to the first-order motion residual information at time t to predict the lost first-order motion residual information at time t+1.

[0024] Among them, the prediction network structure still uses the traditional Transformer encoder-decoder structure. The data input first extracts implicit features through a linear layer and then is sent to the encoder. The encoder part is consistent with the traditional Transformer structure. It consists of 6 identical encoding layers, and each encoding layer consists of a self-attention layer and a feedforward neural network. The decoder part is replaced by a multi-layer perceptron, that is, it consists of a linear layer, an activation function Relu, a linear layer, an activation function Relu, and a linear layer connected in sequence. In terms of data input and output, when the current packet loss is detected, the three frames of motion residual information received before the packet loss are further processed into inter-frame residuals to obtain the second-order motion residual information of a two-frame time window size, which is sent to the prediction network to predict the second-order motion residual information at the next moment, and add it to the motion residual information at the moment before the packet loss to finally predict the packet loss information at the current moment.

[0025] The schematic diagram of the optical flow smoothing method of the present invention is as follows Figure 4 As shown. Based on the first-order optical flow information directly obtained from the source frame to the current frame through the dense motion network, the second-order optical flow information is introduced to better ensure the temporal and spatial consistency between the reconstructed video frames. When the first-order optical flow from the motion frame to the source frame at time t, t-1, and t-2 has been obtained through the dense motion network (t>2), the first-order optical flow at time t-1 and t-2 is subtracted to obtain the second-order optical flow information, and the second-order optical flow information is added to the first-order optical flow at time t-1 to obtain the predicted first-order optical flow at time t, and then the predicted first-order optical flow at time t and the first-order optical flow at time t directly obtained through the dense motion network are respectively assigned a weight of 0.5 and added to obtain the first-order optical flow at time t after smoothing.

Claims

1. An ultra-low bit rate video conferencing compression method based on second-order motion information, characterized in that: The method comprises the following steps: (1) Select a few frames from the video frames and transmit them directly in the form of their own images as source frames, and use the remaining frames as motion frames to extract sparse motion features from the motion frame images; (2) Encoding and compressing the source frame; quantizing the sparse motion features and then performing inter-frame residuals, and encoding the first-order motion residual information obtained through entropy coding; encapsulating the two code streams generated by the source frame and the motion frame into a data packet for transmission; (3) During the transmission process, simulate the packet loss phenomenon in real scenarios and determine whether packet loss occurs. If packet loss occurs, predict the loss information; (4) After determining that no packet loss has occurred or the prediction of packet loss information is completed, the received data packet is divided into two parts: a source frame and first-order motion residual information. Then, sparse motion features are extracted from the source frame, and the first-order motion residual information is accumulated between frames to restore the sparse motion features of the current motion frame. (5) obtaining an occlusion mask of the current motion frame and first-order optical flow information from the source frame to the current frame based on the source frame and its sparse motion features obtained in step (4) and the sparse motion features of the current motion frame; (6) Repeat steps (1) to (5) twice. Since quantizing sparse motion features will cause errors in the first-order optical flow information, starting from the third frame, the first-order optical flow information obtained in the first two frames is used to smooth the optical flow of the current frame through the second-order optical flow information, and then the occlusion mask of the current frame is used to assist in redrawing the optical flow distortion result to reconstruct the current motion frame; In the step (3), if packet loss occurs, the three frames of motion residual information received before the packet loss are further processed into inter-frame residuals to obtain second-order motion residual information of a two-frame time window size, which is then sent to the prediction network to predict the second-order motion residual information at the next moment; the predicted information is added to the first-order motion residual information at the moment before the packet loss to finally predict the packet loss information at the current moment.

2. The ultra-low bit rate video conferencing compression method based on second-order motion information according to claim 1 is characterized in that: In the step (6), a two-frame time window is used as the size, and the first two frames before the current frame are taken to obtain the first-order optical flow information from the source frame to each frame through a dense motion network, and the two are further subtracted to obtain the second-order optical flow information; then the second-order optical flow information is added to the first-order optical flow information of the previous frame of the current frame to obtain a predicted first-order optical flow information of the current frame; then the predicted first-order optical flow information of the current frame and the first-order optical flow information of the current frame directly obtained by the dense motion network are respectively assigned a weight of 0.5 to perform a weighted sum, and finally obtain the optical flow information of the current frame after smoothing.

3. The ultra-low bit rate video conferencing compression method based on second-order motion information according to claim 1, characterized in that: In the step (1), the sparse motion features extracted from each motion frame are composed of 10 key points and a 2x2 affine matrix of each key point.

4. The ultra-low bit rate video conferencing compression method based on second-order motion information according to claim 1, characterized in that: In the step (2), the sparse motion features of each motion frame are first quantized from float32 to int8, and then the inter-frame residual is calculated.

Images