Self-adaptive key frame extraction and video recovery method based on video stream input

Abstract

The invention belongs to the technical field of video semantic communication, and particularly relates to a self-adaptive key frame extraction and video recovery method based on video stream input. The method comprises the following steps: adaptively selecting key frames according to channel conditions for frame information input by a video stream, ignoring non-key frames containing redundant information, realizing real-time and effective key frame extraction and non-key frame insertion back, reducing data stream burden for deep learning to understand video semantics, and avoiding loss of key information at the same time; wherein a video frame recovery simulation module, a frame importance calculator module, an adaptive frame selection module and a video frame recovery module are involved; the method has good compatibility with the current digital communication system. Experimental results show that the method significantly improves the transmission efficiency in the field of video transmission, reduces the channel resource overhead, and promotes the video information semantic reading and multi-scale perception capability of deep learning.

Description

Technical Field

[0001] The present invention belongs to the technical field of semantic communication, and specifically relates to an adaptive key frame extraction and video restoration method. Background Art

[0002] In recent years, with the continuous growth of communication demands, the wireless video data traffic has increased sharply, posing challenges to traditional communication technologies and infrastructure. To meet the video communication demands, a pioneering end-to-end joint source-channel coding method (JSCC) provides an integrated solution for video compression, channel coding, and modulation. Although the video semantic communication system based on JSCC shows the potential to reduce distortion and improve transmission efficiency in theory, the system directly maps the source data to channel symbols, resulting in the arbitrary appearance of constellation points on the constellation diagram, which deviates from the design of current digital communication systems. In addition, most existing studies simulate the channel based on additive white Gaussian noise or other simplified models and cannot capture the complexity of real-world wireless communication, such as multipath effects, signal attenuation, and changing channel conditions. Therefore, the JSCC method has difficulties in robustness and applicability in complex dynamic wireless environments.

[0003] To make video semantic communication more compatible with digital communication systems, an adaptive key frame extraction module and a video restoration module based on the source video data are designed. According to the channel conditions, key frames are adaptively selected based on video semantic information and channel conditions for video compression, and then converted into bitstreams through source coding and channel coding. Subsequently, these bitstreams are mapped to channel symbols to achieve semantic communication. This method can not only reduce distortion but also improve transmission efficiency, and is more in line with the design of digital communication systems. Summary of the Invention

[0004] The purpose of the present invention is to provide an adaptive key frame extraction and video restoration method based on video stream input that can improve communication transmission efficiency and reduce channel resource overhead.

[0005] The adaptive key frame extraction and video restoration method based on video stream input proposed by the present invention includes adaptively selecting key frames according to the channel conditions for the frame information of the video stream input, ignoring non-key frames containing redundant information, and realizing real-time and effective key frame extraction and non-key frame interpolation, reducing the data stream burden for deep learning to understand video semantics, and at the same time avoiding loss of key information; it involves: a video frame restoration simulation module, a frame importance calculator module, an adaptive frame selection module, and a video frame restoration module; the specific steps of the method are as follows:

[0006] (1) For the existing video stream dataset, at the sending end, use the video frame recovery simulation module to simulate the video frame recovery process at the receiving end, and reconstruct the video frames at the receiving end; the video frame recovery simulation module includes an optical flow estimation network and a fusion network module. The optical flow estimation network is used to output the optical flow calculation between two key frames. The frame information around the key frame is represented by marking the motion vectors of the pixel points in each frame relative to the key frame; the video stream data processed by the optical flow estimation network is sent to the continuous fusion network module. The fusion network module fuses the background information extracted at each frame position and the motion vector information output by the optical flow estimation network through continuous multi-layer background information extraction modules, to achieve semantic coding and image reconstruction of the source video stream data;

[0007] (2) Send the semantic understanding image data stream of the same size as the original data stream processed by the multi-layer fusion network to the frame importance calculator module. The frame importance calculator module obtains the β quantitative matrix for evaluating the importance of each frame by calculating the similarity rate between the original image and the reconstructed image;

[0008] (3) The adaptive frame selection module calculates the number of optional key frames according to the channel conditions, and saves bandwidth by preferentially processing key frames to ensure the reliable reception of important information, even under challenging bandwidth conditions;

[0009] (4) After the key frames are selected, they are converted into bitstreams through source coding and channel coding and transmitted to the receiving end;

[0010] (5) After the receiving end receives the bitstream, it obtains the key frames through channel decoding and source decoding. The video frame recovery module uses the optical flow estimation network and the multi-layer fusion network with parameter sharing and the same structure as the video frame recovery simulation module at the sending end to insert non-key frames back and restore the original video.

[0011] Furthermore:

[0012] The described video frame recovery simulation module includes an optical flow estimation network and a multi-layer fusion network. Its basic structure is the cascade of the optical flow estimation network and the multi-layer fusion network. The input is the video stream data, and the output is the simulated and reconstructed video stream; use the multi-layer neural network to simulate the frame recovery process at the receiving end at the sending end, which is convenient for subsequent modules to calculate the frame importance and select key frames.

[0013] The optical flow estimation network in the video frame restoration simulation module and the video frame restoration module is different from the traditional dual optical flow network that estimates the optical flow results of the previous and next frames. The optical flow estimation network of the present invention adopts a strategy from coarse to fine to directly estimate the optical flow of the intermediate frame. In the scenario of fast movement or complex background, simple linear forward and backward flow extraction and interpolation cannot correctly select key frames. Instead, the optical flow estimation network described in the present invention reduces artifact distortion and ambiguity and iteratively improves the optical flow estimation. Its basic principle is to extract rough motion information through a multi-layer convolutional network, gradually restore the high-resolution optical flow field by using upsampling and feature fusion, and adopt residual connection and multi-stage iterative optimization, so that the network can capture the non-linear changes of motion.

[0014] The multi-layer fusion network [1] in the video frame restoration simulation module and the video frame restoration module adopts a prediction framework of multi-layer click-through rate based on context information, uses a cascade of convolution-downsampling-deformation operations to extract the overlapping background information between the previous and next frames of the current frame, and uses a fusion network based on the U-shaped network [2] to realize the information fusion of the background information and the momentum information predicted by the optical flow estimation network on the image at the same position, so as to realize the semantic extraction and image reconstruction of each frame.

[0015] The frame importance calculator module evaluates the importance parameter of each frame in the video by comparing the similarity between the reconstructed video and the original video data. Its input is the video frame and the original video frame reconstructed by the multi-layer fusion network, and the output is the calculation result of the frame importance; for example, if the downstream task attaches more importance to the pixel-level similarity consideration, the pixel-level structural similarity index SSIM should be selected as the calculation result of the frame importance. In the frame importance calculator module, the frame importance parameter β is evaluated by the similarity f n between the reconstructed image Fr n and the original image Fs is . Reference formula:

[0016] β(n)=f is (Fr n ,Fs n ), (1).

[0017] The adaptive frame selection module takes the channel signal-to-noise ratio as the input, and its function is to estimate the key frame ratio according to the level of the signal-to-noise ratio. When the channel condition is poor, it reduces the transmission resources to retransmit the most critical key frames, especially the key frames with a higher importance score β. This adjustment effectively alleviates the cliff effect that will significantly reduce the video quality under extremely poor channel conditions. In the adaptive frame selection module, the quantitative relationship between the key frame ratio ρ and the signal-to-noise ratio (SNR) γ is established as follows:

[0018]

[0019] Among them, a is a parameter estimated using a polynomial regression model.

[0020] The video frame recovery module has a basic network structure including an optical flow estimation network and a multi-layer fusion network that share network parameters and have the same structure as the video frame recovery simulation module at the sending end. Its input is the received video key frame sequence and the non-key frame number matrix. The key frame number matrix is used to mark the number of non-key frames between key frames, facilitating the reconstruction of non-key frames. With the help of the non-key frame number matrix marking the non-key frames between two key frames, the optical flow estimation network calculates the optical flow vectors of non-key frames, and then the lost non-key frames are reconstructed through the multi-layer fusion network.

[0021] The features and advantages of the present invention mainly include:

[0022] The adaptive key frame extractor can dynamically select key frames according to the channel conditions and generate a video sequence recording the positions of these key frames. This selection process is adaptively adjusted based on the content changes between video frames and the quality of the multipath channel. By prioritizing key frames, bandwidth is saved, ensuring reliable reception of important information even under challenging channel conditions. At the same time, the video restorer uses the received key frames to reconstruct the complete video sequence, and a receiver with a relatively simple structure is adopted for practical deployment. Brief Description of the Drawings

[0023] Figure 1 is a retained diagram of the method of the present invention.

[0024] Figure 2 is a flow diagram of the key frame extraction process by the adaptive key frame extractor in the present invention.

[0025] Figure 3 is a structural diagram of the video recovery module in the present invention.

[0026] Figure 4 is the perceptual image block similarity loss between the present invention and the ordinary video transmission method under different signal-to-noise ratios and different modulation methods.

[0027] Figure 5 is the visual structure similarity between the present invention and the ordinary video transmission method under different signal-to-noise ratios and different modulation methods. Detailed Embodiments

[0028] The present invention is further introduced below through embodiments in combination with the drawings.

[0029] It should be noted that the present invention is not limited to the specific embodiments described, and should include improvements, equivalent conversions, and modifications within the scope of the claims.

[0030] Figure 2 is the flowchart of key frame extraction by the adaptive key frame extractor of the present invention, which includes a video frame restoration simulation module (optical flow estimation network, multi-layer fusion network), a frame importance calculator module, and an adaptive frame selection module.

[0031] Figure 3 is the flowchart of the video restoration module of the present invention, which includes an optical flow estimation network and a multi-layer fusion network to realize the interpolation of non-key frames and video restoration.

[0032] In the frame importance calculator module, the frame importance parameter β is evaluated by the similarity f between the reconstructed image Fr n corresponding to each frame and the original image Fs n . The reference formula is: is

[0033] β(n)=f is (Fr n , Fs n )

[0034] In the adaptive frame selection module, the key frame ratio ρ and the signal-to-noise ratio (SNR) γ establish the following quantitative relationship:

[0035]

[0036] where a is the parameter estimated by the polynomial regression model.

[0037] Figure 4 Experimental Example 1: This experiment simulated the loss of perceptual image block similarity (LPIPS) and multi-scale structural similarity (MS-SSIM) under different signal-to-noise ratios (SNR) and different modulation methods (QPSK, 16QAM). The simulation compared the present invention (AKEI) with ordinary video transmission methods, and the results are as Figure 5 and Figure 2 shown. The basic process at the sending end is as Figure 4 shown. The video stream first passes through the video frame restoration simulation module to simulate the reconstruction of video frames, and the frame importance scores are calculated frame by frame for the reconstructed video stream using the key frame calculator. According to the requirements of downstream tasks, MS-SSIM is used as the calculation value of the importance score. Then, according to the channel conditions, the key frame ratio is calculated based on the channel signal-to-noise ratio and the frame importance score, and the key frames are selected for transmission. After the key frames are selected, they are converted into a bit stream through source coding and channel coding and transmitted to the receiving end. After receiving the bit stream, the receiving end obtains the key frames through channel decoding and source decoding. The video frame restoration module uses the optical flow estimation network and the multi-layer fusion network with the same parameters and structure as the video frame restoration simulation module at the sending end to insert the non-key frames back and restore the original video. Figure 4It shows that, compared with ordinary video transmission methods, the method of the present invention can significantly reduce the loss of perceptual image block similarity under low signal-to-noise ratio. For example, when SNR = 5dB and the QPSK modulation method is adopted, the method of the present invention can reduce the loss of perceptual image block similarity from 0.54 to 0.25. Figure 5 It shows that, compared with ordinary video transmission methods, the method of the present invention can significantly improve the visual structure similarity under low signal-to-noise ratio. For example, when SNR = 5dB and the QPSK modulation method is adopted, the method of the present invention can increase the visual structure similarity from 0.51 to 0.78.

[0038] References

[0039] [1] Huang Z, Zhang T, Heng W, et al. RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation[J]. 2020. DOI: 10.48550 / arXiv.2011.06294.

[0040] [2] Ronneberger O. Invited Talk: U-Net Convolutional Networks for Biomedical Image Segmentation[J]. 2017. DOI: 10.1007 / 978-3-662-54345-0_3.

Claims

1. An adaptive key frame extraction and video restoration method based on video stream input, characterized in that, It includes frame information of the video stream input, adaptively selects key frames according to the channel conditions, ignores non-key frames containing redundant information, realizes real-time and effective key frame extraction and non-key frame interpolation, reduces the data stream burden for deep learning to understand video semantics, and avoids losing key information; it involves: a video frame recovery simulation module, a frame importance calculator module, an adaptive frame selection module, and a video frame recovery module; the specific steps of the method are as follows: (1) For the existing video stream dataset, at the sending end, use the video frame recovery simulation module to simulate the video frame recovery process at the receiving end and reconstruct the video frames at the receiving end; the video frame recovery simulation module includes an optical flow estimation network and a fusion network module. The optical flow estimation network is used to output the optical flow calculation between two assumed key frames, and represents the frame information around the key frame by marking the motion vectors of pixel points in each frame relative to the key frame; the video stream data processed by the optical flow estimation network is sent to the continuous fusion network module. The fusion network module fuses the background information extracted at each frame position and the motion vector information output by the optical flow estimation network through continuous multi-layer background information extraction modules, and realizes semantic encoding and image reconstruction of the source video stream data; (2) Send the semantic understanding image data stream of the same size as the original data stream processed by the multi-layer fusion network to the frame importance calculator module. The frame importance calculator module obtains the β quantitative matrix for evaluating the importance of each frame by calculating the similarity ratio between the original image and the reconstructed image; (3) The adaptive frame selection module calculates the number of optional key frames according to the channel conditions, and saves bandwidth by preferentially processing key frames to ensure the reliable reception of important information, even under challenging bandwidth conditions; (4) After the key frames are selected, they are converted into bitstreams through source coding and channel coding and transmitted to the receiving end; (5) After the receiving end receives the bitstream, it obtains the key frames through channel decoding and source decoding. The video frame recovery module uses the optical flow estimation network and the multi-layer fusion network with parameter sharing and the same structure as the video frame recovery simulation module at the sending end to interpolate non-key frames and restore the original video.

2. The method according to claim 1, wherein The video frame recovery simulation module is composed of a cascade of an optical flow estimation network and a multi-layer fusion network, with the input being video stream data and the output being the simulated and reconstructed video stream; using a multi-layer neural network to simulate the frame recovery process at the receiving end at the sending end is convenient for subsequent modules to calculate the frame importance and select key frames; The optical flow estimation network in the video frame recovery simulation module and the video frame recovery module adopts a coarse-to-fine strategy to directly estimate the optical flow of intermediate frames, reduce artifact distortion and ambiguity, and iteratively improve the optical flow estimation; extract rough motion information through a multi-layer convolutional network, use upsampling and feature fusion to gradually restore the high-resolution optical flow field, and adopt residual connections and multi-level iterative optimization to enable the network to capture the non-linear changes of motion; The multi-layer fusion network in the video frame restoration simulation module and the video frame restoration module adopts a multi-layer prediction framework based on click-through rate of context information, uses a cascade of convolution-downsampling-deformation operations to extract the overlapping background information between the previous and the next frames of the current frame, and uses a fusion network based on the U-shaped network to realize the information fusion of the background information and the momentum information predicted by the optical flow estimation network on the image at the same position, so as to realize the semantic extraction and image reconstruction of each frame.

3. The method according to claim 2, wherein In the frame importance calculator module, the importance parameter of each frame in the video is evaluated by comparing the similarity between the reconstructed video and the original video data. Its input is the video frame reconstructed by the multi-layer fusion network and the original video frame, and the output is the calculation result of frame importance; for downstream tasks that pay more attention to the pixel-level similarity consideration, the pixel-level structural similarity index SSIM is selected as the calculation result of frame importance; The frame importance parameter β is calculated from the reconstructed image Fr corresponding to each frame n and the original image Fs n similarity f is evaluated according to the formula: β(n)=f is (Fr n ,Fs n ), (1).

4. The method according to claim 3, wherein The adaptive frame selection module takes the channel signal-to-noise ratio as the input, and its function is to estimate the key frame ratio according to the level of the signal-to-noise ratio. When the channel condition is poor, it reduces the transmission resources to retransmit the most critical key frames, especially the key frames with a higher importance score β; the key frame ratio ρ and the signal-to-noise ratio (SNR) γ have the following quantitative relationship: where a is the parameter estimated by using the polynomial regression model.

5. The method according to claim 3, wherein The input of the video frame restoration module is the received video key frame sequence and the non-key frame number matrix. The non-key frame number matrix is used to mark the number of non-key frames between the key frames, which is convenient for the reconstruction of non-key frames; with the help of the non-key frame number matrix marking the two key frames, the optical flow estimation network calculates the optical flow vectors of the non-key frames, and then reconstructs the lost non-key frames through the multi-layer fusion network.

Images