Video motion magnification method based on improved autoencoder network

Abstract

The present invention relates to a video motion magnification method based on an improved autoencoder network, which uses the improved autoencoder network to magnify subtle changes in a video. The method comprises: decomposing and processing video data, using two consecutive frames of images before and after the decomposition as the input of an encoder; using the encoder to extract the shape features of the two consecutive frames of images as the input of an amplifier; using the amplifier to amplify the pixel displacement difference of the shape features of the two frames of images to obtain the amplified shape features; using a decoder to upsample the texture features of the previous frame image, and combining the texture features with the amplified shape features to obtain an amplified frame and output it. The method of the present invention realizes the perfect fusion of shape and texture features in a video motion magnified image, reduces the brightness, color, and texture losses of video motion magnification, and preserves shallow feature information.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular relates to a video motion amplification method based on an improved autoencoding network. Background Art

[0002] Most studies tend to focus on information that is easily observed by the naked eye, but some important changes in the video cannot be processed by computers. In response to this research status, the MIT team proposed a video motion magnification technology called "motion microscopy" that can assist in obtaining this important information. Video motion magnification technology can magnify subtle changes in the video to a level that can be observed by the naked eye, such as detecting blood circulation and micro-expression recognition. However, as the video magnification factor continues to increase, the video image information will be overly blurred and have a large number of noise artifacts, which may cause the outline of the moving object to disappear and change the authenticity of the movement.

[0003] The earliest research technology for video motion magnification was based on fluid mechanics. This method achieved motion magnification by clustering and tracking feature points. The disadvantage is that it requires manual operation throughout the process and has high computational complexity. Later, the Euler magnification method for image processing appeared. It pioneered the use of pixel changes to enhance and magnify the area of ​​interest, providing a method and basis for later video image magnification.

[0004] As the research on deep learning methods gradually deepens, a deep learning-based method is proposed. This method solves the problem of traditional manual filter design, but there are still image distortion, artifacts and contour deformation for videos with high magnification. Although the image texture features are not amplified in the autoencoder network, there are still problems such as local color changes, reduced image brightness and contrast. At the same time, there is a color loss phenomenon in the fusion of image texture features and shape features, which is likely to cause a checkerboard effect in severe cases. Summary of the invention

[0005] The technical problem addressed by the present invention is that the existing deep learning-based video motion magnification methods have image distortion, artifacts and contour deformation, and the fusion of image texture features and shape features has color loss and even a checkerboard effect.

[0006] The purpose of the present invention is to solve the above-mentioned problems and provide a video motion magnification method based on an improved autoencoder network. The autoencoder network is improved by adding deformable convolution and channel attention mechanism. The characteristics of the channel attention mechanism are used to assign different weight information to areas with inconsistent color and texture changes, so as to better integrate shallow image texture features with deep structural information; deformable convolution is used to extract changing shape features in the image, broaden the convolution field of view, adaptively adjust the shape of the convolution kernel, integrate more spatial structural information, improve the denoising ability of the network, and ensure that the network extracts more accurate dynamic change feature information in the image; the improved autoencoder network is used to amplify subtle changes in the video and minimize brightness, color and texture losses.

[0007] The technical solution of the present invention is a video motion amplification method based on an improved autoencoder network, which uses an improved autoencoder network to amplify subtle changes in a video. The improved autoencoder network includes an encoder, an amplifier and a decoder. The encoder uses a channel attention mechanism and deformable convolution to improve the ability of the autoencoder network to extract image texture and shape features. The encoder of the improved autoencoder network includes a texture feature extraction unit and a shape feature extraction unit. The texture feature extraction unit uses a channel attention mechanism, and the shape feature extraction unit includes a convolution layer, a deformable convolution and a residual block.

[0008] The video motion magnification method comprises the following steps:

[0009] Step 1: Decompose the video data and use the two consecutive frames of images as the input of the encoder;

[0010] Step 2: Use the encoder to extract the shape features of two consecutive frames of images as the input of the amplifier;

[0011] Step 3: Amplify the pixel displacement difference of the shape features of the two frames of images by using an amplifier to obtain the amplified shape features;

[0012] Step 4: Use the decoder to upsample the texture features of the previous frame image, and combine the texture features with the enlarged shape features to obtain the enlarged frame and output it;

[0013] Step 4.1: Use the decoder to upsample the texture features of the previous frame image output by the encoder to ensure that the texture features and the enlarged shape features have the same size;

[0014] Step 4.2: Fuse the upsampled texture features with the enlarged shape features to obtain the enlarged frame.

[0015] Preferably, the decoder of the improved autoencoder network includes a feature fusion layer, multiple residual blocks, an upsampling layer and a channel attention mechanism.

[0016] Furthermore, step 3 includes the following sub-steps:

[0017] Step 3.1: Calculate the two frames of image I extracted by the encoder A ,I B Shape feature F SA 、F SB The difference between A Represents the first frame of two consecutive frames, I B Indicates the second frame image, F SA represents the shape feature of the first frame image, F SB Represents the shape features of the second frame image;

[0018] Step 3.2: Multiply the pixel displacement difference of the shape feature by the amplification factor α to perform motion amplification to obtain the amplified shape feature;

[0019] Step 3.3: Combine the enlarged shape features with image I A Shape feature F SA Superposition to obtain the enlarged shape features

[0020]

[0021] Where G m () represents the entire video motion magnification operation process. This formula represents a nonlinear calculation process, the purpose of which is to reduce noise interference. h() represents the residual kernel function, and g() represents the convolution operation function of shape feature differences.

[0022] Preferably, in step 4, the decoder performs a convolution operation on the texture features obtained by upsampling and the amplified shape features after fusion, and upsamples the convolved feature map to the original input size, and introduces a residual block to reduce the loss in upsampling.

[0023] Preferably, during the training process of the improved autoencoder network, a loss function Loss is introduced to ensure that the improved autoencoder network can accurately learn the specific details of motion amplification during the training process, and interference is added in the loss function calculation for regularization. The calculation formula of the loss function Loss is as follows

[0024] Loss = L1(I M ,I M ′)+λ(L1(I A ,I C )+L1(I B ,I M )+L1(I B ,I M ′))

[0025] Where I Mrepresents the enlarged frame image output by the improved autoencoder network, I M ′ represents the real magnified image, I C Indicates that the image with interference noise is added during the training process to improve the network's anti-interference ability and optimize network performance. The λ parameter represents the weight information used to standardize network training, making the network training effect more conducive to expressing the law of shape change. L1() represents the L1 norm loss function, L1(I M ,I M ′) represents the enlarged image I output by the improved autoencoder network M Compared with the real enlarged image I M ′, L1(I A ,I C ) represents the first frame image I A With image I C Texture loss, L1(I B ,I M ) represents the second frame image I B With the enlarged frame image I M Texture loss, L1(I B ,I M ′) represents the second frame image I B Compared with the real enlarged image I M 'Loss of motion.

[0026] Compared with the prior art, the beneficial effects of the present invention include:

[0027] 1) The method of the present invention realizes the perfect fusion of shape and texture features in video motion magnification images, reduces the brightness, color and texture losses of video motion magnification, and preserves shallow feature information.

[0028] 2) The improved autoencoder network model of the present invention expands the representation ability of neural networks. The channel attention mechanism of the improved autoencoder network integrates features at different levels and adaptively learns different weight information from the attention module, and dynamically adjusts the weight according to the changing feature information in the output image.

[0029] 3) The deformable convolution adaptive adjustment mechanism of the improved autoencoder network model of the present invention enables shallow feature information to adaptively flow from the upsampled part to higher-level features, better maintaining the effectiveness of image features. Traditional spatially invariant convolution kernels can cause image texture destruction and excessive noise artifacts. The improved autoencoder network model of the present invention uses deformable convolution to replace traditional convolution methods, dynamically expands the receptive field, can integrate more spatial structure information, and further improve the performance of the denoising network.

[0030] 4) The improved autoencoding network of the present invention has lower demand for training data, and a data set with a small amount of sample data can achieve the training effect, which further verifies the feasibility of the improved autoencoding network.

[0031] 5) The improved autoencoder network model of the present invention has strong generalization ability. Compared with the traditional model, the model shows better video motion amplification effect for different test video sequences. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0033] Figure 1 Schematic diagram of the structure of an improved autoencoding network according to an embodiment of the present invention.

[0034] Figure 2 Schematic diagram of the structure of an encoder according to an embodiment of the present invention.

[0035] Figure 3 Schematic diagram of the structure of a decoder according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] like Figure 1 As shown in FIG, the improved autoencoder network of the example includes an encoder, an amplifier and a decoder. The encoder includes a texture feature extraction unit and a shape feature extraction unit. The texture feature extraction unit adopts a channel attention mechanism. The shape feature extraction unit includes a convolution layer, a deformable convolution and a residual block. Figure 2 shown.

[0037] like Figure 3 As shown, the decoder of the improved autoencoder network includes a feature fusion layer, 9 residual blocks, an upsampling layer, a channel attention unit, and a convolutional layer connected in sequence.

[0038] The video motion magnification method based on the improved autoencoder network includes the following steps:

[0039] Step 1: Decompose the video data and use the two consecutive frames of image I A ,I B As the input of the encoder, I A Represents the first frame of two consecutive frames, I B Indicates the second frame image.

[0040] Step 2: To ensure that the encoder can extract more accurate image feature information, feature information of different dimensions is used inside the encoder to represent the texture features F of the image. T and shape feature F SThe channel attention module is added to extract texture features. Different weight information is assigned to areas with inconsistent color and texture changes by using the characteristics of the channel attention mechanism, so as to better integrate shallow image texture features F T Fusion with deep structural information. Use deformable convolution to extract the changing shape features in the image F S , broaden the convolution field of view, adaptively adjust the shape of the convolution kernel, integrate more spatial structure information, improve the network's denoising ability, and ensure that the network extracts more accurate dynamic change feature information in the image.

[0041] Deformable convolution adds an offset Δb to each point based on the original convolution. n In order to ensure that the generated pixels correspond to the actual points in the feature map, the bilinear interpolation method is used to ensure the accuracy of the acquired pixels. The bilinear interpolation method is an extension of the interpolation of two variables. The core idea is to perform linear interpolation in the horizontal direction and the vertical direction respectively.

[0042] Extract image I through encoder A Feature map represents the representation space of the feature map, C represents the number of channels of the feature map, H and W represent the height and width of the feature map respectively. The channel attention mechanism can construct an operation unit for any input information, and the operation process is:

[0043]

[0044] where F tr Represents the convolution operation on the feature map, U A Represents the output feature set.

[0045] To ensure that the network can increase the extraction of useful information and accurately model the dependencies between channels, we first consider the output feature U A The global spatial information is compressed into a channel descriptor, that is, the shape of the feature map is changed from C×H×W to C×1×1, and a global average pooling layer is used to realize channel statistics. C It is represented by two convolutional layers and activated by Sigmoid function and Rule function. The calculation formula is as follows:

[0046] CA c =σ(Conv(δ(Conv(g c ))))

[0047]

[0048] CA CRepresents the weight information of each channel of the image, σ() represents the Sigmoid function, δ() represents the Rule activation function, and finally inputs each element into F A With channel CA C The weights are multiplied pixel by pixel to obtain the texture feature information F extracted by the attention mechanism. TA .

[0049] Step 3: Apply the pixel change relationship in the amplifier and extract the image I through the calculation encoder A ,I B Shape feature F SA 、F SB The difference is obtained by multiplying the pixel displacement difference by the magnification factor α to achieve the effect of motion amplification, and finally the amplified difference is superimposed back to F SA , and get the enlarged shape features

[0050]

[0051] Where G m () is a nonlinear function, which represents the entire video motion amplification operation process, the purpose of which is to reduce the interference of noise. h() represents the residual kernel function, and g() represents the convolution function that performs convolution operation on the shape feature difference.

[0052] Step 4: The feature fusion part includes the texture feature part output by the encoder and the shape feature part output by the amplifier. A ,I B Extract shape features F SA 、F SB , after the amplifier operation, the output is amplified and superimposed back to the original shape Input image I A After the channel attention operation, the output is the extracted texture feature information F TA To ensure the consistency of feature map size during fusion, F TA Upsampling is Finally, the sampled texture features are converted into The shape features after enlargement Fusion The fused feature map The image is restored to its original size through residual and upsampling operations to output the enlarged image I M .

[0053] The improved autoencoder network of the present invention introduces a loss function during iterative training to ensure that the network can accurately learn the specific operational details of motion magnification during the training process, and adds interference to regularize the loss function calculation. The data of the training data set includes the original two frames of image I A and I B , training set image I with interference noise added C And the real magnified image I m The loss function of the present invention is based on the L1 norm loss function, and the image I is calculated and generated. M Compared with the real enlarged image I m ′, denoted as L1(I M ,I M ′), use L1(I A ,I C ) represents image I C Relative I A Texture loss, L1(I B ,I M ) represents the second frame image I B With the enlarged frame image I M Texture loss, L1(I B ,I M ′) represents the second frame image I B Compared with the real enlarged image I M The total loss function is as shown in the formula:

[0054] Loss = L1(I M ,I M ′)+λ(L1(I A ,I C )+L1(I B ,I M )+L1(I B ,I M ′))

[0055] Wherein L1() represents the L1 norm loss function, λ is a weight parameter, which is used to represent the weight information of the interference term in training, so that the network training effect is more conducive to expressing the law of shape change. In the embodiment, λ is set to 0.1.

Claims

1. A video motion magnification method based on an improved autoencoder network, characterized in that: An improved autoencoder network is used to amplify subtle changes in a video. The improved autoencoder network includes an encoder, an amplifier, and a decoder. The encoder uses a channel attention mechanism and deformable convolution to improve the autoencoder network's ability to extract image texture and shape features. The method includes the following steps: Step 1: Decompose the video data and use the two consecutive frames of images as the input of the encoder; Step 2: Use the encoder to extract the shape features of two consecutive frames of images as the input of the amplifier; Step 3: Amplify the pixel displacement difference of the shape features of the two frames of images by using an amplifier to obtain the amplified shape features; Step 3 includes the following sub-steps: Step 3.1: Calculate the two frames extracted by the encoder , Shape characteristics , The difference between Represents the first frame of two consecutive frames of images, Represents the second frame image, represents the shape features of the first frame image, Represents the shape features of the second frame image; Step 3.2: Multiply the pixel displacement difference of the shape feature by the magnification factor Perform motion magnification to obtain magnified shape features; Step 3.3: Combine the enlarged shape features with the image Shape characteristics Superposition to obtain the enlarged shape features , ； In the formula Represents the nonlinear calculation function for the entire video motion amplification, represents the residual kernel function, Convolution function representing the difference in shape features; Step 4: Use the decoder to upsample the texture features of the previous frame image, and combine the texture features with the enlarged shape features to obtain the enlarged frame and output it; Step 4.1: Use the decoder to upsample the texture features of the previous frame image output by the encoder to ensure that the texture features and the enlarged shape features have the same size; Step 4.2: Combine the upsampled texture features with the enlarged shape features to obtain the enlarged frame.

2. The video motion magnification method according to claim 1, characterized in that: The encoder of the improved autoencoder network includes a texture feature extraction unit and a shape feature extraction unit. The texture feature extraction unit adopts a channel attention mechanism, and uses the characteristics of the channel attention mechanism to assign different weight information to the inconsistent color and texture change areas; The shape feature extraction unit includes a convolution layer, a deformable convolution and a residual block. The shape feature extraction unit uses deformable convolution to extract the changing shape features in the image, broaden the convolution field of view, adaptively adjust the shape of the convolution kernel, and extract the dynamically changing feature information in the image.

3. The video motion magnification method according to claim 2, characterized in that: The decoder of the improved autoencoder network includes a feature fusion layer, multiple residual blocks, an upsampling layer, and a channel attention mechanism.

4. The video motion magnification method according to claim 1, characterized in that: In step 4, the decoder performs a convolution operation on the texture features obtained by upsampling and the amplified shape features, and upsamples the convolved feature map to the original input size, and introduces a residual block to reduce the loss in upsampling.

5. The video motion magnification method according to any one of claims 1 to 4, characterized in that: Improve the training process of the autoencoder network and introduce the loss function Ensure that the improved autoencoder network can accurately learn the specific details of motion amplification during training, and add interference to regularize the loss function calculation. The calculation formula is as follows ； In the formula Represents the enlarged frame image output by the improved autoencoder network, Indicates the image captured by real magnification. represents the training image with interference noise added, is the weight parameter; express Norm loss function, Zoomed-in image showing the output of the improved autoencoder network Compared with the real enlarged image The loss between Indicates the first frame image With image The texture loss, Represents the second frame image With enlarged frame image The texture loss, Represents the second frame image Image taken with real magnification loss of movement.

Images