Inter-frame error concealment method, device and medium based on variational autoencoder

Abstract

The present invention discloses an inter-frame error concealment method, device and medium based on a variational autoencoder, which relates to the field of image processing technology, and includes the steps of: extracting the inter-frame images before and after the lost frame in the video compression process, and extracting the features of the inter-frame images before and after; outputting the mean and standard deviation of the latent variables by mapping the extracted features to a latent space; generating latent variables by sampling from a standard normal distribution, combining the output mean and standard deviation, and mapping the latent variables back to the original image space to obtain a generated image; taking minimizing the reconstruction error as the goal, measuring the difference between the generated image and the real image based on a difference loss function, and optimizing the parameters of the variational autoencoder; and mapping the inter-frame images before and after to the latent space and reconstructing them into a filling frame by using the variational autoencoder after parameter optimization. The present invention can effectively improve the video quality and reduce the visual distortion introduced by inter-frame image errors.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an inter-frame error concealment method, device and medium based on a variational autoencoder. Background Art

[0002] VVC (Versatile Video Coding) is a new generation of video coding standard, which aims to replace the HEVC coding standard, with significantly improved compression efficiency and higher coding flexibility. However, in the video compression process, video errors are very likely to occur, resulting in the overall loss of some inter-frame images, so the video error concealment (VEC) technology was born.

[0003] Video error concealment is used to deal with data loss or corruption problems encountered during video transmission and encoding. This technology is particularly important for improving video quality, especially under poor network conditions. The basic goal of video error concealment is to restore or simulate the original video content as much as possible when data packets are lost or errors occur, so as to provide a continuous and visually pleasing video stream. The following are several common video error concealment methods: (1) Retransmission and forward error correction coding (FEC): Retransmission: When a data packet transmission error is found, the lost or damaged data packet is resent. Forward error correction coding: The sender adds redundant information to the data packet, which can be used by the receiver to correct small errors or recover lost data. (2) Intra-frame / inter-frame residual replication: Part of the data (such as residual information) of the surrounding frames is copied to the position of the lost frame to fill the lost part. This method is simple and effective, but it may lead to a decrease in visual quality. (3) Motion compensation: Use inter-frame motion estimation technology to predict and fill the lost frame part based on the motion information between adjacent frames. This method can restore video frames more finely and maintain high visual quality.

[0004] Although video error concealment technology has made significant progress, it still faces some challenges: error concealment needs to be completed within a limited time to meet the needs of real-time transmission; efficient algorithm design to reduce the consumption of computing resources is very necessary, especially for mobile devices; and the ability to automatically adjust the concealment strategy according to different network conditions and video content to achieve the best recovery effect. Summary of the invention

[0005] In order to improve the quality and reliability of video and reduce the impact of frame loss caused by video compression, the present invention proposes an inter-frame error concealment method based on a variational autoencoder, wherein the variational autoencoder includes the functions of an encoder and a decoder, and comprises the steps of:

[0006] S1: Extract the images before and after the frames lost during the video compression process, and extract the features of the images before and after the frames by reducing the spatial dimension and increasing the feature depth;

[0007] S2: Output the mean and standard deviation of the latent variables by mapping the extracted features into the latent space;

[0008] S3: Generate latent variables by sampling from the standard normal distribution, combining the mean and standard deviation of the output, and map the latent variables back to the original image space to obtain the generated image;

[0009] S4: With the goal of minimizing the reconstruction error, the difference between the generated image and the real image is measured based on the difference loss function, and the parameters of the variational autoencoder are optimized;

[0010] S5: Through the variational autoencoder with optimized parameters, the images between the previous and next frames are mapped to the latent space and reconstructed into the filled frames.

[0011] Furthermore, in the step S1, the extracted previous and next frame images also include pre-processing of normalization and enhancement operations.

[0012] Furthermore, the S1 step and the S2 step are represented by the following neural network expression:

[0013]

[0014] In the formula, is the mean of the latent variable, is the standard deviation of the latent variable, The previous inter-frame image used to replace the lost frame and the next inter-frame image Neural network converted to the mean and standard deviation of the latent variable, including convolutional layers, pooling layers, and fully connected layers.

[0015] Furthermore, in the step S1, feature extraction of the previous and next frame images is performed through multi-layer convolution and pooling operations.

[0016] Furthermore, in the step S2, the output of the last convolutional layer is flattened through the fully connected layer and connected to the two output nodes to obtain the mean and standard deviation of the latent variable, which is expressed as the following formula:

[0017]

[0018] In the formula, is the mean of the latent variable, is the standard deviation of the latent variable, is the output of the last convolutional layer, and is the weight matrix, and is the bias term.

[0019] Furthermore, in the step S3, a convolutional neural network is used to perform a mapping conversion from latent variables to generated images, and a decoding function of a variational autoencoder is implemented, which is expressed as the following formula:

[0020]

[0021] In the formula, z is the latent variable, is the mean of the latent variable, is the standard deviation of the latent variable, is element-wise multiplication, To generate the image, A neural network is used to reconstruct the image by mapping the latent variable z of the input layer back to the original data space through the hidden layer.

[0022] Furthermore, the difference loss function includes reconstruction loss, KL divergence loss and error concealment loss, which are respectively expressed as the following formulas:

[0023]

[0024] In the formula, is the reconstruction loss defined by the pyramid structure, s is each scale in the pyramid, is the weight of scale s, and are the versions of the generated image and the real image at scale s, respectively. To generate images for measurement and real images The mean square error function of the difference between is the KL divergence loss, and are the standard deviation and mean of the j-th dimension in the latent space, respectively, and J is the number of dimensions of the latent space; is the error concealment loss used to characterize the difference between the compressed video frame and the decompressed video frame, To compress video frames, Decompress the video frames, is the expected value; For the overall loss, is the weight coefficient used to weigh the error concealment loss and other losses.

[0025] The present invention also includes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the inter-frame error concealment method based on a variational autoencoder are implemented.

[0026] Also included is a device for processing data, comprising:

[0027] a memory having a computer program stored thereon;

[0028] A processor is used to execute the computer program in the memory to implement the steps of the inter-frame error concealment method based on variational autoencoder.

[0029] Compared with the prior art, the present invention has at least the following beneficial effects:

[0030] The present invention discloses an inter-frame error hiding method, device and medium based on a variational autoencoder, which applies the variational autoencoder to video error hiding, utilizes the motion consistency of images between adjacent frames, combines the bidirectional prediction capability of the VVC coding standard, learns the distribution law of latent variables through the variational autoencoder, and adjusts the parameters, thereby mapping the previous and next frames to the latent space and reconstructing them into padding frames, so as to minimize the reconstruction error and maintain the continuity and smoothness of the latent space, effectively improve the video quality, and reduce the visual distortion introduced by inter-frame image errors. The method is particularly suitable for video transmission and storage applications that require high-quality image restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A step diagram of an inter-frame error concealment method based on variational autoencoder. DETAILED DESCRIPTION

[0032] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to these embodiments.

[0033] Video error concealment technology plays an important role in a variety of application scenarios, including but not limited to: maintaining smooth video playback and reducing freezes when network conditions are poor; ensuring clear video communication even in the case of network packet loss, which is critical for both business and personal communication; and applications in scenarios such as telemedicine and security monitoring, which require video transmission to be highly reliable and stable. The Variational Autoencoder (VAE) is a deep learning model that combines the concept of autoencoders with variational Bayesian methods. VAE is mainly used for generative modeling, that is, learning the probability distribution of data and being able to generate new samples from it.

[0034] Compared to the autoencoder under traditional unsupervised learning technology, VAE not only focuses on how to reconstruct the input data, but also tries to learn the distribution of latent variables. This enables VAE to sample from the latent space (the low-dimensional representation space used in the generative model, in which the original high-dimensional data is converted into a set of lower-dimensional variables, which are usually called latent variables or hidden variables) to generate new data instances. Based on this concept, we can learn the data probability distribution of the adjacent frames step by step based on the characteristics of great motion coherence and similarity between adjacent frames, so as to generate fill-in frames to make up for the lost frames between the two.

[0035] Therefore, in order to better deal with the frame loss problem that may occur in video compression transmission under the VVC coding standard, the present invention proposes to apply VAE to video error concealment technology, such as Figure 1 As shown, a method for inter-frame error concealment based on variational autoencoder is proposed, including the following steps:

[0036] S1: Extract the images before and after the frames lost during the video compression process, and extract the features of the images before and after the frames by reducing the spatial dimension and increasing the feature depth;

[0037] S2: Output the mean and standard deviation of the latent variables by mapping the extracted features into the latent space;

[0038] S3: Generate latent variables by sampling from the standard normal distribution, combining the mean and standard deviation of the output, and map the latent variables back to the original image space to obtain the generated image;

[0039] S4: With the goal of minimizing the reconstruction error, the difference between the generated image and the real image is measured based on the difference loss function, and the parameters of the variational autoencoder are optimized;

[0040] S5: Through the variational autoencoder with optimized parameters, the images between the previous and next frames are mapped to the latent space and reconstructed into the filled frames.

[0041] Based on the principle of VAE mentioned above, here we first need to find the location of the lost frames during the video compression process and extract the lost frames The previous interframe image and the next inter-frame image , and perform preprocessing, such as normalizing the image to a suitable scale and performing necessary enhancement operations (such as rotation, cropping, etc.). It should be noted here that since we need to extract the inter-frame images of one frame before and after the lost frame, the present invention is only applicable to video coding standards with a bidirectional prediction mode, such as VCC.

[0042] In the VCC error concealment method, an encoder needs to be designed to receive and lose frames. Adjacent previous and next frame images are mapped to the latent space. To this end, the present invention designs a convolutional neural network including a convolutional layer, a pooling layer and a fully connected layer to implement the encoding function. Among them, multi-layer convolution and pooling operations are used to extract the features of the input frame by gradually reducing the spatial dimension and deepening the feature depth, and the extracted features are mapped to the latent space. Here, we use a fully connected layer to map the features, which is to flatten the output of the last convolutional layer and connect it to two output nodes, represented as and .

[0043]

[0044] In the formula, is the mean of the latent variable, is the standard deviation of the latent variable, The inter-frame image before the frame that will be lost and the next inter-frame image Transformed into a neural network with the mean and standard deviation of the latent variables. At this point, assuming that the output of the last convolutional layer is ,but:

[0045]

[0046] In the formula, and is the weight matrix, and is the bias term.

[0047] In this way, based on the obtained mean and standard deviation, we can reparameterize the variational autoencoder by Sampling from the middle, combining the mean and standard deviation obtained after encoding, generates the latent variable z:

[0048]

[0049] in, is element-wise multiplication.

[0050] Here, we design The neural network implements the encoder encoding function and learns the mapping function so that the input frame can be represented as the parameters of the Gaussian distribution in the latent space.

[0051] Then, we design another convolutional neural network to implement the decoding function, receive the latent variable z and generate the padded frame . First, the latent variable z is mapped to the initial feature map size before decoding through a fully connected layer. This step can usually be regarded as the starting point of decoding, thus starting to generate output. Then, deconvolution (also known as transposed convolution or upsampling) operations are used to gradually expand the spatial dimensions of the feature map and gradually reduce the depth of the features. These operations help to transform the low-dimensional latent variable z into a high-dimensional image similar to the original. Finally, the output layer generates the same image as the lost frame. Generate an image that corresponds as closely as possible to the real image , the generated image decoded from the latent variable z can be expressed as:

[0052]

[0053] Here, we design The neural network implements the decoding function of the decoder by learning to map the latent variable z back to the original image space, with the goal of minimizing the reconstruction error, and using the pixel-level difference loss function to measure the difference between the generated image and the real image. Here, the difference loss function includes reconstruction loss, KL divergence loss, and error concealment loss.

[0054] Among them, the reconstruction loss is used to measure the generated image after decoding and reconstruction. With real image The difference between the two, in this invention, a pyramid structure is used to define the loss, which is expressed as:

[0055]

[0056] In the formula, is the reconstruction loss, s is each scale in the pyramid, is the weight of scale s, and are the versions of the generated image and the real image at scale s, respectively. To generate images for measurement and real images This multi-scale loss method can effectively consider the features and structures of images at different scales, thereby improving the quality and visual perception of generated images.

[0057] KL Divergence Loss is used to ensure that the learned latent variable distribution is close to the standard normal distribution to promote the generation of more continuous and realistic filling results:

[0058]

[0059] In the formula, is the KL divergence loss, and are the standard deviation and mean of the j-th dimension in the latent space, respectively, and J is the dimension of the latent space, which usually corresponds to the dimension of the latent space of the encoded output.

[0060] The reconstructed generated image Compress to get compressed video frame , for compressed video frames Decompress and get the decompressed video frame , then the error concealment loss is the compressed video frame and decompress video frames The difference between:

[0061]

[0062] in, is the expected value.

[0063] Combining reconstruction loss, KL divergence loss and error concealment loss, the overall loss is expressed as:

[0064]

[0065] In the formula, For the overall loss, is the weight coefficient used to weigh the error concealment loss and other losses. By minimizing the overall loss , thereby optimizing the parameters in the variational autoencoder.

[0066] In this way, the variational autoencoder with optimized parameters can reconstruct the input data and fill in the lost frames according to the learned distribution of latent variables. That is, through the variational autoencoder with optimized parameters, the images between the previous and next frames are mapped to the latent space and reconstructed into the filled frames.

[0067] The present invention also includes a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the inter-frame error concealment method based on a variational autoencoder are implemented.

[0068] Also included is a device for processing data, comprising:

[0069] a memory having a computer program stored thereon;

[0070] A processor is used to execute the computer program in the memory to implement the steps of the inter-frame error concealment method based on variational autoencoder.

[0071] In summary, the inter-frame error concealment method, device and medium based on variational autoencoder described in the present invention apply the variational autoencoder to video error concealment, utilize the motion consistency characteristics of images between adjacent frames, combine the bidirectional prediction capability of the VVC coding standard, learn the distribution law of latent variables through the variational autoencoder, and adjust the parameters, so as to map the previous and next frames to the latent space and then reconstruct them into filling frames, so as to minimize the reconstruction error and maintain the continuity and smoothness of the latent space, effectively improve the video quality, and reduce the visual distortion introduced by inter-frame image errors. It is particularly suitable for video transmission and storage applications that require high-quality image restoration.

[0072] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0073] In addition, in the present invention, descriptions such as "first", "second", "one", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0074] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0075] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Claims

1. An inter-frame error concealment method based on a variational autoencoder, wherein the variational autoencoder includes the functions of an encoder and a decoder, characterized in that: The error concealment method comprises the steps of: S1: Extract the images before and after the frames lost during the video compression process, and extract the features of the images before and after the frames by reducing the spatial dimension and increasing the feature depth; S2: Output the mean and standard deviation of the latent variables by mapping the extracted features into the latent space; S3: Generate latent variables by sampling from the standard normal distribution, combining the mean and standard deviation of the output, and map the latent variables back to the original image space to obtain the generated image; S4: With the goal of minimizing the reconstruction error, the difference between the generated image and the real image is measured based on the difference loss function, and the parameters of the variational autoencoder are optimized; S5: Through the variational autoencoder with optimized parameters, the images between the previous and next frames are mapped to the latent space and reconstructed into the filled frames.

2. The inter-frame error concealment method based on variational autoencoder according to claim 1, characterized in that: In the step S1, the extracted previous and next frame images also include pre-processing of normalization and enhancement operations.

3. The inter-frame error concealment method based on variational autoencoder according to claim 1, characterized in that: The S1 step and the S2 step are represented by the following neural network expressions: In the formula, is the mean of the latent variable, is the standard deviation of the latent variable, The previous inter-frame image used to replace the lost frame and the next inter-frame image Neural network converted to the mean and standard deviation of the latent variable, including convolutional layers, pooling layers, and fully connected layers.

4. The inter-frame error concealment method based on variational autoencoder according to claim 3, characterized in that: In the step S1, feature extraction of the previous and next frame images is performed through multi-layer convolution and pooling operations.

5. The inter-frame error concealment method based on variational autoencoder according to claim 3, characterized in that: In the S2 step, the output of the last convolutional layer is flattened through the fully connected layer and connected to the two output nodes to obtain the mean and standard deviation of the latent variable, which is expressed as the following formula: In the formula, is the mean of the latent variable, is the standard deviation of the latent variable, is the output of the last convolutional layer, and is the weight matrix, and is the bias term.

6. The inter-frame error concealment method based on variational autoencoder according to claim 1, characterized in that: In the step S3, a convolutional neural network is used to perform a mapping conversion between latent variables and generated images, and the decoding function of the variational autoencoder is implemented, which is expressed as the following formula: In the formula, z is the latent variable, is the mean of the latent variable, is the standard deviation of the latent variable, is element-wise multiplication, To generate the image, is a neural network used to reconstruct the image by mapping the latent variable z of the input layer back to the original data space through the hidden layer. are the parameters sampled from a standard normal distribution.

7. The inter-frame error concealment method based on variational autoencoder according to claim 1, characterized in that: The difference loss function includes reconstruction loss, KL divergence loss and error concealment loss, which are respectively expressed as the following formulas: In the formula, is the reconstruction loss defined by the pyramid structure, s is each scale in the pyramid, is the weight of scale s, and are the versions of the generated image and the real image at scale s, respectively. To generate images for measurement and real images The mean square error function of the difference between is the KL divergence loss, and are the standard deviation and mean of the j-th dimension in the latent space, respectively, and J is the number of dimensions of the latent space; is the error concealment loss used to characterize the difference between the compressed video frame and the decompressed video frame, To compress video frames, Decompress the video frames, is the expected value; For the overall loss, is the weight coefficient used to weigh the error concealment loss and other losses.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the error concealment method according to any one of claims 1 to 7 are implemented.

9. A device for processing data, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the error concealment method according to any one of claims 1 to 7.

Images