Audio-assisted facial video coding method

Abstract

Disclosed in the present invention is an audio-assisted facial video coding method, comprising the following steps: step 1: using a video coder to code an input video to obtain a corresponding bitstream, and recording the bitstream as Vbit; step 2: using an audio coder to code an input audio to obtain a corresponding bitstream, and recording the bitstream as Abit; step 3: using an audio decoder to decode the audio bitstream Abit to generate an audio file; and step 4: a video decoder performing decoding using the audio file and the video bitstream Vbit to generate a video file for playback. In the present invention, the semantic relevance between an audio and a video is utilized during coding / decoding, so that such relevance can be fully utilized, thereby reducing bitstreams to be transmitted, and further improving the subjective and objective quality of facial videos.

Description

Audio-assisted face video coding method TECHNICAL FIELD

[0001] The present application belongs to the technical field of source coding, and particularly relates to an audio-assisted face video coding method. BACKGROUND

[0002] In recent years, the growing demand for video conferencing / telephony relies heavily on the effective coding and transmission of face videos. In the past few decades, in order to improve the coding performance, people have developed many video coding standards. The most advanced video coding standard at present, VVC (Versatile video coding), has improved the coding efficiency by 30-50% compared with HEVC (High Efficiency video coding). With the development of deep learning, video coding methods based on deep learning have also been widely studied. A large number of studies have shown that integrating neural networks into traditional coding frameworks and replacing some of the sub-modules can significantly improve coding efficiency. At the same time, there are also studies that use deep learning technology in the post-processing stage of video coding to reduce the distortion caused by coding and improve video quality. In addition, end-to-end video coding schemes have also received extensive research and attention, aiming to tap more performance gains through end-to-end design. At present, these schemes can be comparable in performance to traditional coding schemes. The above several methods are general video coding methods, and there is no requirement for video content, and any content video can be coded. However, when it comes to face videos, the situation is more special. The rich prior knowledge contained in face videos, such as the stability of the face structure and the short-time invariance of the appearance of the person, especially the characteristics that the motion of the video is mainly concentrated in the face area of the person, are expected to further improve the coding efficiency of face videos.

[0003] There is a face video coding method in the prior art (publication number: CN114422795A), which uses strong prior knowledge of face videos to achieve good coding efficiency at ultra-low code rate. This method encodes the key frame by a traditional coding method, and extracts compact features from the remaining non-key frames. At the decoding end, the powerful generation ability of the generation model is used to generate the current frame image in combination with the texture information of the key frame and the motion information of the compact features. However, the audio information is not involved in this method.

[0004] There are methods in the prior art that use audio information to assist in post-processing of human face video. The method uses audio information to assist video enhancement at the decoding end to reduce artifacts in decoded video at low bit rates. Although these methods use the correlation between audio and video to improve reconstruction quality, these methods are a post-processing method that uses audio information to assist in quality enhancement of decoded video and do not affect the encoding and decoding process of human face video. The code rate depends on the selected video encoding method and cannot fully utilize the correlation between audio and video.

[0005] Current human face video encoding technology mainly focuses on video modalities and rarely considers the correlation of audio modalities. However, in daily scenarios, audio modalities and video modalities usually coexist and there is a correlation between them. The existing audio information assisted human face video post-processing scheme uses audio information to assist in video denoising and does not affect the encoding and decoding process, which cannot fully utilize the correlation between audio and video. SUMMARY

[0006] To overcome the deficiencies of the prior art, the purpose of the present application is to provide an audio-assisted human face video encoding method that utilizes the semantic correlation between audio and video during encoding and decoding, which can more fully utilize this correlation, not only reducing the required code stream, but also improving the subjective (DISTS and LPIPS) and objective (PSNR and SSIM) quality of human face video.

[0007] To achieve the above purpose, the technical solution adopted by the present application is:

[0008] An audio-assisted human face video encoding method, comprising the following steps:

[0009] Step 1: For the input video, use a video encoder to encode and obtain a video code stream Vbit;

[0010] Step 2: For the input audio, use an audio encoder to encode and obtain an audio code stream Abit; The audio is collected simultaneously with the video, and the main content in the audio is the voice of the person in the input video of step 1;

[0011] Step 3: Use an audio decoder to decode the audio code stream Abit to generate an audio file;

[0012] Step 4: The video decoder uses the audio file and the video code stream Vbit to decode and generate a video file for playback.

[0013] In step 1, the video is divided into two parts, key frames and non-key frames. The first frame is selected as the key frame, and the remaining frames are selected as the non-key frame. If the video duration is long, every time interval (such as several seconds), the frame played at the current time is selected as the key frame, and the remaining frames are selected as the non-key frame.

[0014] The key frame is encoded using an image / video encoding method (such as VVC, HEVC, etc.), and the code stream is obtained.

[0015] For non-key frames, first, the feature extraction network is used to extract the feature.

[0016] The extracted non-key frame feature is encoded, and the non-key frame feature code stream is obtained.

[0017] The difference between the extracted feature and the reconstructed feature of the non-key frame is calculated, and the residual is quantized to an integer. Then the quantized result is converted to binary code, and finally the binary code is encoded using entropy coding to obtain the non-key frame feature code stream.

[0018] The key frame code stream and the non-key frame feature code stream jointly constitute the video code stream Vbit.

[0019] In step 2, the audio encoder uses an audio encoding method (such as advanced audio coding, linear predictive coding, etc.) to obtain the audio code stream Abit.

[0020] In step 3, first, the audio code stream Abit is decoded using an audio decoding method to obtain an audio file. Then, the audio file is passed through an audio feature extraction module to extract the audio features related to the video.

[0021] The audio feature extraction module extracts audio features. In the video stream, a segment of audio file (such as 0.4s) corresponding to the current decoding frame is selected for each frame, and then the audio features are calculated. The type of audio feature is not limited, which can be common mel frequency cepstral coefficient (MFCC) or perceptual linear prediction coefficient (PLP) etc.

[0022] The following is an embodiment for calculating MFCC audio features. The audio is pre-emphasized, framed, windowed, power spectrum calculated, mel filter used, log transformed and DCT transformed to obtain MFCC (Mel Frequency Cepstral Coefficient) features.

[0023] Specifically, the pre-emphasis is to pass the speech signal through a high-pass filter to boost the high-frequency part, then take N sampling points as a frame, divide the sampling points of the audio into multiple frames, where there is an overlap of part of the sampling points between adjacent frames, then multiply each frame by a Hamming window to reduce spectral leakage, the processed speech frame is subjected to fast Fourier transform to obtain the energy distribution on the spectrum, then take the modulus square to obtain the power spectrum, and finally pass the energy spectrum through a filter bank of Mel scale, calculate the logarithmic energy of the output of the filter bank, and then substitute the logarithmic energy into the discrete cosine transform to obtain the MFCC feature. With the help of a machine learning model, the audio features are further processed to obtain the audio features Afeat related to the video;

[0024] In step 4, for the video code stream, the key frame and non-key frame features are respectively reconstructed, and the key frame, non-key frame features and extracted audio features are used to generate the corresponding optical flow map and occlusion map through an audio-guided optical flow map generation module;

[0025] Based on the key frame, audio feature, optical flow map and occlusion map, the non-key frame is generated through an audio-assisted inter-frame generation module;

[0026] Finally, the key frame and the generated non-key frame are combined to obtain a complete video file.

[0027] The audio-guided optical flow map generation module establishes the motion relationship between the key frame and the current frame, and since audio information is used for guidance, more accurate and robust optical flow map and occlusion map can be obtained, which is beneficial to improving the quality of the final video;

[0028] First, a feature extraction network is used to extract a representation feature feat1 from the key frame, and a representation feature feat2 of the current non-key frame is reconstructed; the structure and parameters of the feature extraction network are not limited, for example: an Hourglass network based on a combination of a convolutional neural network and a generalized division normalization activation function can be used.

[0029] Secondly, a sparse optical flow map is established between feat1 and feat2 using an optical flow algorithm, and the optical flow field is used to perform preliminary image deformation estimation on the down-sampled key frame to obtain a down-sampled warped image warpimg.

[0030] Thirdly, a difference operation is performed on feat1 and feat2 to obtain a feature difference, denoted as featd.

[0031] The warped image warpimg and featd are input into an optical flow estimation network for dense motion estimation.

[0032] The output of the optical flow estimation network is an optical flow map and an occlusion map, representing the motion relationship between the key frame and the current frame, and used in the next module.

[0033] Further, additional audio information is beneficial to obtain more accurate and robust optical flow maps and occlusion maps, so that a part of structure for interaction between video features and audio features can be added to the optical flow estimation network. The interaction method of the two modalities is not limited, and methods such as multi-modal feature fusion and multi-modal feature guidance can be used. For example, an Hourglass network based on a combination of a convolutional neural network and a generalized division normalization activation function can be used, and some convolutional layers can be replaced with adjustable weight convolutional layers.

[0034] An embodiment of multi-modal feature guidance based on adjustable weight convolutional layers uses the extracted audio features Afeat to dynamically adjust the convolutional layer weight ω ijk , so as to realize audio-guided optical flow map generation.

[0035] The calculation formula of the specific adjustment process is shown in the following formula (1):

[0036] where i, j and k represent the input channel number, the output channel and the spatial position of the convolution kernel respectively, ε is a constant, which prevents the denominator from being 0, Afeat i is the audio feature of the i-th channel, ω ijk is the weight of the convolution kernel, and the output of the optical flow estimation network is an optical flow map and an occlusion map, representing the motion relationship between the key frame and the current frame, and used in the next module.

[0037] Specifically, the optical flow map is the offset direction and size of the pixels between the key frame and the current frame, and the occlusion map is a feature map with a value between 0 and 1, and the value represents the confidence of the pixel at that point.

[0038] The audio-assisted inter-frame generation module uses the key frame and the optical flow map and the occlusion map obtained from the previous module to generate the current frame under the assistance of audio.

[0039] The texture features, including the face and part of the background, are extracted from the key frame, and then the texture features are deformed using the optical flow map.

[0040] Since there is motion between the key frame and the current frame, not all pixels of the current frame can be found in the key frame, so the deformed features need to be multiplied by the occlusion map to represent the confidence of the deformed features, and the video features multiplied by the occlusion map are denoted as feato.

[0041] The video features feato are input into the generation model to obtain the reconstructed video frame.

[0042] Further, the additional audio information is beneficial to improve the quality of the reconstructed video frame. In order to further improve the quality of the reconstructed video frame, in some embodiments, the audio feature Afeat and the video feature feato can be fused and input into the generation model to obtain the reconstructed video frame. Such fusion is beneficial to further utilize the audio feature and improve the quality of the final video. The type of fusion method is not limited, and direct splicing, attention mechanism fusion, matrix operation fusion, feature projection fusion, etc. can be used.

[0043] An implementation based on self-attention mechanism fusion first up-samples the audio feature Afeat to the same size as the video feature, and through a convolution layer, as K and V, and the video feature feato after the convolution layer as Q. V T The transpose matrix of V is represented as Q T The transpose matrix of Q is represented as V. Wherein softmax(·) represents the softmax function, and the purpose is attention map normalization. Then the Attenfeature(Q,K,V) is calculated by the following formula (2). T K)V T (2)

[0044] Finally, the feature Attenfeature(Q,K,V) extracted by the attention mechanism is spliced with the original video feature feato, and then input into the generation network to obtain the finally generated picture.

[0045] The beneficial effects of the present application are:

[0046] The present application is different from the previous audio-assisted post-processing scheme, and the semantic correlation of audio and video is used in the process of coding and decoding, and the correlation is more efficiently used to assist coding and decoding.

[0047] The present application uses audio features in the process of inferring non-key frames. And since the network is trained end-to-end, the audio features can affect the encoding and decoding process at the same time. In addition, the specially designed loss function contains the code stream size, so that the network in the encoding end will learn by itself to reduce the repeated transmission of related information to effectively compress the code stream, so as to make the loss function decrease.

[0048] The present application not only reduces the repeated transmission of related information to reduce the transmission bandwidth, but also improves the subjective and objective quality of the video, and brings a smoother and clearer audio-visual experience. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a flow chart of the multi-modal joint coding method of the present application.

[0050] Figure 2 is a schematic diagram of the audio-assisted face video coding method of the present application. DETAILED DESCRIPTION

[0051] The application will be described in further detail below with reference to the drawings.

[0052] The technical solution of the application is shown in the following figure:

[0053] Figure 1 gives the framework of the proposed audio-aided face video coding scheme.

[0054] For the input video and audio, a video encoder and an audio encoder are used for encoding, respectively, to obtain the corresponding code streams, denoted as Vbit and Abit.

[0055] At the decoding end, once the Vbit and Abit data streams are received, the audio decoder is used to decode the audio code stream Abit to generate audio files, since the decoding of Vbit needs to use the features extracted from the audio.

[0056] These decoded audio files can not only be directly used for playing, but also play an important role in the decoding process of the video files. Because in the video decoder, the features extracted from the audio are needed in the process of generating non-key frames from the non-key frame features. Subsequently, the video decoder uses these audio files and the video code stream Vbit to decode and generate video files for playing.

[0057] Figure 2 is a specific embodiment of Figure 1.

[0058] The encoding end mainly includes a video encoder and an audio encoder.

[0059] The audio encoder uses an audio encoding method (such as advanced audio coding, linear predictive coding, etc.) to obtain the audio code stream Abit.

[0060] The video is divided into two parts, key frames and non-key frames. The first frame is selected as the key frame, and the remaining frames are selected as the non-key frames. If the video duration is long, every time interval, the frame played at the current time is selected as the key frame, and the remaining frames are selected as the non-key frames.

[0061] If the quality of the video generated is desired to be good, the selection interval of the key frame can be selected to be a short time.

[0062] Further, an image / video encoding method (such as VVC, HEVC, etc.) is used to encode the key frame to obtain the code stream;

[0063] For the non-key frame, a feature extraction network is first used to extract the representative features. The structure and parameters of the feature extraction network are not limited, for example: an Hourglass network based on a combination of a convolutional neural network and a generalized division normalization activation function, etc.

[0064] The extracted non-key frame features are encoded to obtain a feature code stream of the non-key frame;

[0065] The features are inter-frame predicted, quantized and entropy encoded to obtain a code stream. The specific feature compression process is: difference operation is performed between the extracted features of the non-key frame and the reconstructed features to obtain a residual, and then the residual is quantized into an integer. Then the quantized result is converted into a binary code, and finally the binary code is encoded by using entropy coding to obtain the feature code stream of the non-key frame.

[0066] The key frame code stream and the non-key frame feature code stream jointly constitute a video code stream Vbit.

[0067] The audio code stream Abit and the video code stream Vbit can be combined for network transmission or storage.

[0068] In the decoding end, the audio code stream Abit is first decoded by using an audio decoding method to obtain an audio file;

[0069] Subsequently, the audio file will pass through a special audio feature extraction module to extract video-related audio features;

[0070] For the video code stream, the key frame and the non-key frame features are reconstructed, and then the key frame, the non-key frame features and the previously extracted audio features are used to generate a corresponding optical flow map and an occlusion map through an audio-guided optical flow map generation module;

[0071] Based on the key frame, the audio features, the optical flow map and the occlusion map, the non-key frame is generated through an audio-assisted inter-frame generation module;

[0072] Finally, the key frame and the generated non-key frame are combined to obtain a complete video file.

[0073] In the audio feature extraction module, the audio feature extraction: in the video stream, a segment of audio file corresponding to the current decoding frame is selected for each frame, and then the features of the audio are calculated. The type of audio features is not limited, which can be common mel frequency cepstral coefficient (MFCC) or perceptual linear prediction coefficient (PLP), etc.; the audio features are further processed by using a machine learning model to obtain video-related audio features Afeat; the structure and parameters of the machine learning model used for audio feature processing are not limited, for example: a long short-term memory network (LSTM) network with a gating unit can be used, etc. There are many features in the audio, but not all of them are related to the video, so the video-related audio features Afeat are extracted from the audio and combined with the video features in the subsequent operation.

[0074] The audio-guided optical flow map generation module.

[0075] The role of the module is to establish the motion relationship between the key frame and the current frame;

[0076] First, the feature extraction network is used to extract the representation feature feat1 from the key frame, and the representation feature feat2 of the current non-key frame is reconstructed, and the structure and parameters of the feature extraction network are not limited, and the feature extraction network for extracting the non-key frame feature at the encoding end can be referred to;

[0077] Then, a sparse optical flow map is established between feat1 and feat2 using an optical flow algorithm, and the optical flow field is used to perform preliminary image deformation estimation on the down-sampled key frame to obtain the down-sampled warped image warpimg;

[0078] Then, the difference operation is performed on feat1 and feat2 to obtain the feature difference, which is denoted as featd;

[0079] The warped image warpimg and featd are input into the optical flow estimation network for dense motion estimation;

[0080] The structure and parameters of the optical flow estimation network are not limited, and can be set according to actual needs, for example: an Hourglass network based on a combination of a convolutional neural network and a generalized division normalization activation function can be used;

[0081] Further, additional audio information is beneficial to obtain more accurate and robust optical flow maps and occlusion maps, so that a part of structure for interaction between video features and audio features can be added to the optical flow estimation network. The interaction method of the two modalities is not limited, and methods such as multi-modal feature fusion and multi-modal feature guidance can be used. For example: an Hourglass network based on a combination of a convolutional neural network and a generalized division normalization activation function can be used, and some convolutional layers can be replaced with adjustable weight convolutional layers;

[0082] The following is an embodiment of multi-modal feature guidance based on an adjustable weight convolutional layer, which uses the extracted audio feature Afeat to dynamically adjust the convolutional layer weight ω ijk , so as to realize audio-guided optical flow map generation.

[0083] The calculation formula of the specific adjustment process is shown in the following formula (1):

[0084] Where i, j and k represent the input channel number, the output channel and the spatial position of the convolution kernel respectively, and ε is a constant that prevents the denominator from being 0. Afeat i is the audio feature of the i-th channel, and ω ijkare weights of the convolution kernel, and the output of the optical flow estimation network is an optical flow map and an occlusion map, which represent the motion relationship between the key frame and the current frame and are used in the next module. Specifically, the optical flow map is the offset direction and size of a pixel between the key frame and the current frame, and the occlusion map represents the reliability of the pixel at the point.

[0085] The audio-assisted interframe generation module, which is used to reconstruct the video frame.

[0086] This module extracts texture features, including a face and part of the background, from the key frame, and then deforms the texture features using the optical flow map;

[0087] Since there is motion between the key frame and the current frame, not all pixels of the current frame can be found in the key frame, so it is necessary to multiply the deformed features by the occlusion map to represent the reliability of the deformed features, and the video features multiplied by the occlusion map are denoted as feato

[0088] The video features feato are input into the generation model to obtain the reconstructed video frame.

[0089] Further, additional audio information is beneficial to improve the quality of the reconstructed video frame. In order to further improve the quality of the reconstructed video frame, in some embodiments, the audio features Afeat and the video features feato can be fused and then input into the generation model. Such fusion is beneficial to further utilize the audio features to improve the quality of the final video. The type of fusion method is not limited, and direct splicing, attention mechanism fusion, matrix operation fusion, feature projection fusion, etc. can be used;

[0090] The following is an embodiment based on self-attention mechanism fusion. First, the audio features Afeat are upsampled to the same size as the video features, and are input into the convolution layer as K and V, while the video features feato are input into the convolution layer as Q after being convolved. V T represents the transpose matrix of V, Q T represents the transpose matrix of Q. Wherein softmax(·) represents the softmax function, and the purpose is to normalize the attention map. Then, the Attenfeature(Q, K, V) is calculated by the following formula (2). T K)V T (2)

[0091] Finally, the features Attenfeature(Q, K, V) extracted by the attention mechanism are spliced with the original video features feato, and then input into the generation network to reconstruct the video frame.

[0092] The above is the process during testing. During the network training stage, the entire network is trained end-to-end, so that the audio features simultaneously affect the encoding process and the decoding process, not only improving the generation quality of the non-key frames, but also reducing the code stream of the non-key frame features. The loss function It can be designed in the following form:

[0093] wherein Rate(·) represents the number of bits required for conversion to a binary code stream, which is usually obtained by estimation during the training stage. Quality(·,·) measures the quality of reconstruction, which can be measured using a perceptual loss or a subjective quality indicator such as DISTS. and G are the loss functions of the generative adversarial network, representing the loss functions of the generator and the discriminator, respectively.

[0094] The present application saves 9.88%, 7.22%, 8.00%, and 8.26% compared to the existing single-modal generative face video encoding method when using the indicators PSNR, SSIM, DISTS, and LPIPS as quality indicators to calculate BD-rate.

[0095] The present application saves 3.35%, 2.90%, 19.02%, and 12.58% compared to the existing audio-assisted face video post-processing method (the encoding and decoding method uses a single-modal generative face video encoding method) when using the indicators PSNR, SSIM, DISTS, and LPIPS as quality indicators to calculate BD-rate.

[0096] The present application first encodes and decodes the same video using two encoding methods to obtain the encoded code stream size and the corresponding decoded video. The two encoding methods can use different configuration parameters to obtain multiple different code stream sizes and corresponding decoded videos. Then, the quality indicators (PSNR, SSIM, DISTS, and LPIPS) of the decoded videos are calculated, and the calculated indicators and the corresponding code stream sizes are plotted into a code stream-quality curve graph, thereby calculating the BD-rate.​

Claims

1. An audio-assisted face video encoding method, characterized in that, The method comprises the following steps; Step 1: encode the input video using a video encoder to obtain a video bitstream Vbit; Step 2: encode the input audio using an audio encoder to obtain an audio bitstream Abit; Step 3: decode the audio bitstream Abit using an audio decoder to generate an audio file; Step 4: decode the video bitstream Vbit using the audio file and the video bitstream Vbit to generate a video file for playing.

2. The method of claim 1, wherein, In the step 1, the video is divided into key frames and non-key frames, the first frame is selected as the key frame, and the remaining frames are selected as the non-key frames; if the video duration is long, every time interval, the frame played at the current time is selected as the key frame, and the remaining frames are selected as the non-key frames.

3. The method of claim 2, wherein, The key frame is encoded using an image / video encoding method to obtain a bitstream; For the non-key frame, first, a feature extraction network is used to extract a representation feature, and the feature is encoded to obtain a bitstream; Specifically, a difference operation is performed between the extracted feature of the non-key frame and the reconstructed feature to obtain a residual, then the residual is quantized into an integer, then the quantized result is converted into a binary code, and finally the binary code is encoded using an entropy encoding to obtain a feature bitstream of the non-key frame; The key frame bitstream and the non-key frame feature bitstream jointly constitute the video bitstream Vbit.

4. The method of claim 3, wherein, In the step 2, the audio encoder encodes using an audio encoding method to obtain an audio bitstream Abit.

5. The method of claim 4, wherein, In the step 3, first, the audio bitstream Abit is decoded using an audio decoding method to obtain an audio file; then, the audio file passes through an audio feature extraction module to extract a video-related audio feature; The audio feature extraction module extracts the audio feature, in the video stream, a length of audio file corresponding to the frame being decoded is selected every frame, and then the feature of the audio is calculated, and the type of the audio feature is not limited.

6. The audio-assisted human face video encoding method of claim 5, wherein, The audio feature is further processed by means of a machine learning model to obtain a video-related audio feature Afeat; the structure and parameters of the machine learning model used for audio feature processing are not limited.

7. The method of claim 6, wherein, In the step 4, for the video bitstream, the key frame and the non-key frame feature are reconstructed respectively, and the key frame, the non-key frame feature and the extracted audio feature are used to generate a corresponding optical flow graph and an occlusion graph through an audio-guided optical flow graph generation module; Then, based on the key frame, the audio feature, the optical flow graph and the occlusion graph, a non-key frame is generated through an audio-assisted inter-frame generation module; Finally, the key frame and the generated non-key frame are combined to obtain a complete video file.

8. The method of claim 7, wherein, The audio-guided optical flow graph generation module establishes a motion relationship between the key frame and the current frame; First, a feature extraction network is used to extract a representation feature feat1 from the key frame, and a representation feature feat2 of the current non-key frame is reconstructed; Secondly, a sparse optical flow graph is established between feat1 and feat2 using an optical flow algorithm, and a down-sampled warped image warpimg is obtained by using the optical flow field to preliminarily estimate the image deformation of the down-sampled key frame. Again, the feat1 and feat2 are subjected to difference operation, to obtain feature difference, and recorded as featd; The warp image warpimg and the featd are input into an optical flow estimation network to perform dense motion estimation.

9. The method of claim 7, wherein, The structure and parameters of the optical flow estimation network are not limited, and are set according to actual needs; Or using Hourglass network based on convolutional neural network and generalized division normalization activation function combination, and replacing part of the convolutional layer with an adjustable weight convolutional layer; The adjustable weight convolution layer is used to dynamically adjust the convolution layer weight ω by using the extracted audio feature Afeat ijk Thus, the audio-guided optical flow map generation is realized. The calculation formula of the specific adjustment process is shown in the following formula (1): where i, j and k represent the input channel number, output channel and spatial position of the convolution kernel respectively, ε is a constant, Afeat i is the audio feature of the i-th channel, ω ijk is the weight of the convolution kernel, and the output result of the optical flow estimation network is an optical flow map and an occlusion map, representing the motion relationship between the key frame and the current frame and being used in the next module.

10. The method of claim 9, wherein, The optical flow map is the offset direction and size of the pixel between the key frame and the current frame, and the occlusion map represents the reliability of the pixel at the point; The audio-assisted interframe generation module uses the key frame and the optical flow map and the occlusion map obtained by the last module to generate the current frame under the assistance of audio; The texture features are extracted from the key frame, including the face and part of the background, and then the optical flow map is used to deform the texture features; The deformed features are multiplied by the occlusion map to represent the reliability of the deformed features, and the video features multiplied by the occlusion map are recorded as feato; The video features feato are input into a generation model to obtain a reconstructed video frame. or fuse the audio feature Afeatand the video feature feato using a self-attention mechanism. Specifically, the audio feature Afeatis upsampled to the same size as the video feature and passed through a convolutional layer as K and V, while the video feature feato is passed through a convolutional layer as Q, V T denotes the transpose of V, Q T denotes the transpose of Q, where softmax(·) denotes a softmax function, and then calculated by the following equation (2); Attenfeature(Q, K, V) = softmax(Q T K)V T (2) The attention mechanism extracted features Attenfeature(Q, K, V) and the original video features feato are spliced, and then sent into a GAN to obtain the finally generated picture.

Images