Video motion magnification method based on transformer network

Abstract

The application discloses a video motion magnification method based on a Transformer network, and steps of the method comprise the following steps: 1, extracting motion representation by using a proposed motion representation encoder; 2, calculating the difference of the motion representation to obtain motion changes; 3, performing nonlinear magnification by using a proposed motion magnification conversion module to obtain magnified motion representation; 4, fusing the magnified motion representation with the features of the motion representation of a query frame; 5, obtaining fine magnified fusion features by using a proposed multi-refinement decoder; and 6, reconstructing a high-quality motion magnification image. The application can quickly and accurately magnify and visualize the micro motion in a video which is difficult for the human eye to observe, and the proposed Transformer network can effectively suppress problems such as ringing artifacts, noise and motion blur existing in the motion magnification video, so that a high-quality motion magnification video can be generated.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing and computer vision, and mainly relates to a video motion magnification method based on a Transformer network. BACKGROUND

[0002] The main purpose of video motion magnification technology is to break through the limitations of human visual perception resolution and reveal the small motion hidden in the macro world. Because this technology can serve as a computer-aided "eye" and vividly describe the small motion in the real world that is difficult for human vision to perceive, it has attracted widespread attention in many fields. For example: modal analysis and vibration mode visualization of instruments, bridges, etc. to detect the structural characteristics of the target; in the medical field, the video is magnified and enhanced for the heart and blood vessels to assist doctors in better diagnosis and treatment of diseases; perception enhancement of face recognition and micro-motion recognition, etc.

[0003] Thanks to the rapid development of computer vision and deep learning, the video motion magnification method based on deep learning has made great progress with its advanced generalization ability and robustness, but still faces the following problems:

[0004] Firstly, the existing methods are usually based on CNN network architecture, which limits the ability of the network model to learn long pixel dependencies, resulting in partial information loss during the magnification feature transformation process, and the phenomenon of ringing artifacts and motion noise.

[0005] For example, in 2018, Tae-Hyun Oh et al. published an article entitled "Learning-based video motion magnification" in the top international conference Proceedings of the European Conference on Computer Vision. This method uses an end-to-end network architecture based on CNN to magnify local motion changes. However, due to the limitation of CNN in ignoring the correlation between local and global features, the magnified video usually contains chaotic motion noise generated by magnification, which introduces unnecessary magnification.

[0006] Secondly, the current video motion magnification method based on deep learning cannot well adapt to various scenes and is prone to distortion and blur when dealing with complex scenes, which is not conducive to practical scene applications.

[0007] For example, in 2021, Biagio Brattoli et al. published an article entitled "Unsupervised behaviour analysis and magnification (UBAM) using deep learning" in the international top journal Nature Machine Intelligence. This paper proposes a video motion magnification method based on the difference in different human body postures using unsupervised learning, which uses an auto-encoder to separate the appearance and posture features of different individuals walking, and uses them to generate magnified postures. However, the application scene of this method is relatively single, and the magnified video generated is often not clear enough, so it cannot have good generalization ability and robustness. SUMMARY

[0008] To solve the above problems of the prior art, a video motion magnification method based on a Transformer network is proposed to generate high-quality motion magnified videos and effectively improve the generalization ability of the model to various scenes.

[0009] To solve the above problems, the following method scheme is adopted:

[0010] The video motion magnification method based on the Transformer network has the following characteristics:

[0011] Step 1, obtain continuous frame images containing micro motion changes, and take the first frame image as the reference frame, denoted as ; take the second frame image as the query frame, denoted as ; perform color perturbation on the query frame to obtain the color-perturbed query frame, denoted as ; the corresponding real magnified frame of the reference frame and the query frame is denoted as ; the magnification factor of the real magnified frame is denoted as , wherein represents the real number field, represents the height of the image, represents the width of the image, and 3 represents the number of channels of the image;

[0012] Step 2, construct a Transformer network, including a motion representation encoder, a motion magnification conversion module, and a multi-refinement decoder.

[0013] Define the current training number as , and initialize . Let the first The parameters of the Transformer network during training are: ;

[0014] Step 3: The motion representation encoder uses equation (1) to process the reference frames respectively. Query Frame query frames with color perturbation Feature extraction is performed, and the corresponding result is obtained. Reference features during training Query features and disturbance characteristics :

[0015] (1)

[0016] In equation (1), Indicates the first During training, the kernel size is... Convolution operation, Dimensions representing features;

[0017] Step 4: The motion representation encoder uses equation (2) for the reference feature. Processing yields the first... Query vector during training Key vector Sum value vector :

[0018] (2)

[0019] In equation (2), Indicates the first Pointwise convolution operation during training Indicates the first Depthwise convolution operation during training; Presentation layer normalization operation, This indicates a tiling operation;

[0020] Will , and Divide along the channel dimension of the feature respectively Size, thus obtaining , and ;in, Indicates the first During the training session The query vector of the attention head. Indicates the first During the training session The key vector of the attention head, denotes the value vector of the th attention head at the th training;

[0021] Step 5, the motion representation encoder calculates the motion representation of the th training using formula (3) :

[0022] (3)

[0023] In formula (3), denotes the activation function, denotes the transpose operation, denotes the temperature parameter to be learned at the th training;

[0024] Step 6, the motion representation encoder calculates the cross-covariance attention between , and of the th attention head at the th training using formula (4) :

[0025] (4)

[0026] Step 7, the motion representation encoder processes the cross-covariance attention of the n attention heads using formula (5) to obtain the feature mapping of the th training :

[0027] (5)

[0028] In formula (5), denotes the concatenation operation of the n attention heads;

[0029] Step 8, the motion representation encoder processes the feature mapping using formula (6) to generate the motion representation of the reference frame at the th training :

[0030] (6)

[0031] In formula (6), denotes the gating mechanism of the double-path dot product, and has:

[0032] ​​​​ (7)

[0033] In equation (7), The expansion factor represents the feature dimension. express Nonlinear activation function; , They represent the first During the training iteration, Pointwise and Depthwise convolution operations are performed on the first path. , They represent the first During the second training iteration, Pointwise and Depthwise convolution operations are performed on the second path. It represents the Hadamardi (or Hadama) stack;

[0034] Step 9: Following the process of steps 4-8, and using the motion representation encoder to share weights, process the query features respectively. and disturbance characteristics The process is performed to obtain the query frame. motion representation Query frames with color perturbation motion representation ;

[0035] Step 10: The motion amplification conversion module calculates the first step using equation (8). Two frames of motion representation during training and The difference :

[0036] (8)

[0037] Step 11: The motion amplification conversion module calculates the first step using equation (9). Training time difference Magnified motion representation :

[0038] (9)

[0039] Step 12, the multiple refinement decoder refines the first... The amplified motion during the second training session represents With query frame motion representation Feature concatenation is performed along the channel dimension to obtain the first... Amplified fusion features during training ;

[0040] Step 13: The multiple refinement decoders are stacked. The amplified fusion features are processed layer by layer in the manner of a Transformer layer to obtain fine amplified fusion features ;

[0041] Step 14, the multi-refinement decoder reconstructs the fine amplified fusion features by using formula (10) to obtain high-quality amplified frames in the t-th training : :

[0042] (10)

[0043] In formula (10), represents the sub-pixel accuracy upsampling operation in the t-th training;

[0044] Step 15, the error of the Transformer network in the t-th training is calculated by using formula (11) : :

[0045] (11)

[0046] In formula (11), is a hyperparameter for balancing the two loss errors, represents the Charbonnier loss function;

[0047] Step 16, it is judged whether the error in the t-th training converges; if the error converges, it indicates that the parameters of the Transformer network in the t-th training are optimal parameters , and the Transformer network corresponding to the optimal parameters is used as the final model for generating video motion amplification; on the contrary, the Transformer network continues to perform the t+1-th gradient back propagation, and the parameters of the Transformer network are updated , and the parameters of the Transformer network in the t-th training are obtained After that, step 17 is performed.

[0048] Step 17, t+1 is assigned to t, and the process returns to step 3 for sequential execution.

[0049] ​​​​​​​​​The electronic device comprises a memory and a processor, and the memory is used for storing a program supporting the processor to execute the video motion amplification method, and the processor is configured to execute the program stored in the memory.

[0050] The computer readable storage medium stores a computer program, and when the computer program is run by a processor, the steps of the video motion amplification method are executed.

[0051] Compared with the prior art, the present application has the following beneficial effects:

[0052] 1、The present application introduces an advanced Transformer module that aggregates local-global context to maintain high-resolution feature maps, solving the problem of ringing artifacts in existing video motion amplification techniques.

[0053] 2、The present application designs an efficient motion amplification conversion module to cope with different scene motion changes, which acts on each pixel position and supports cross-channel nonlinear feature interaction to enhance feature transformation representation learning, thereby improving the information integrity and accuracy of amplified features.

[0054] 3、The present application proposes a multi-refinement decoder stacked by multiple layers of the proposed Transformer module, which aims to guide attention calculation in a hierarchical manner to refine the final amplified feature attention map and remove spatial noise layer by layer to give more weight to important regions of motion amplification, thereby effectively reducing ringing artifacts, noise and other problems in the reconstruction process of amplified features. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 The network model of the present application is shown in the figure;

[0056] Figure 2 The network model training diagram of the present application is shown in the figure;

[0057] Figure 3 The multi-scale attention mechanism feature fusion diagram of the present application is shown in the figure. DETAILED DESCRIPTION

[0058] In this embodiment, a video motion magnification method based on a Transformer network can quickly and accurately magnify and visualize the micro-motions in a video that are difficult for the human eye to observe, effectively solving the problems of ringing artifacts, noise, motion blur, and the like in the magnified video, thereby generating a high-quality motion magnified video. Specifically, the following steps are taken:

[0059] Step 1, obtain continuous frame images containing micro-motion changes, and take the first frame image as the reference frame, denoted as ; take the second frame image as the query frame, denoted as ; perform color perturbation on the query frame to obtain the color-perturbed query frame, denoted as ; take the reference frame and the query frame as the corresponding real magnified frame, denoted as ; take the magnification factor of the real magnified frame as , where represents the real number field, represents the height of the image, represents the width of the image, and 3 represents the number of channels of the image;

[0060] Step 2, as shown in Figure 1 , a Transformer network is constructed, mainly including three parts: a motion representation encoder, a motion magnification conversion module, and a multi-refinement decoder;

[0061] Define the current training number as , and initialize . Let the parameters of the Transformer network at the th training be , use the Adam optimizer for training, and set the learning rate to 0.0001;

[0062] Step 3, as shown in Figure 2 , the motion representation encoder uses formula (1) to extract features from the reference frame , the query frame , and the color-perturbed query frame , respectively, to obtain the reference feature , the query feature , and the perturbed feature at the th training:

[0063] (1)

[0064] In formula (1), represents the During training, the kernel size is... Convolution operation, Dimensions representing features;

[0065] Step 4: The motion representation encoder uses equation (2) for the reference feature. Processing is performed to obtain the first... Query vector during training Key vector Sum value vector :

[0066] (2)

[0067] In equation (2), Indicates the first The kernel size during training is 1. Pointwise convolution operations are used to effectively aggregate pixel-level cross-channel local features; Indicates the first The kernel size during training is 1. The depthwise convolution operation is used to encode spatial context information of the feature channel direction; Presentation layer normalization operation, This indicates a tiling operation;

[0068] Will , and Divide along the channel dimension of the feature respectively Size, thus obtaining , and ;in, Indicates the first During the training session The query vector of the attention head. Indicates the first During the training session The key vector of the attention head, Indicates the first During the training session The value vector of the attention head;

[0069] Step 5: The motion representation encoder calculates the first step using equation (3). During the training session Transposed attention map of attention head :

[0070] (3)

[0071] In equation (3), express activation function, denotes a transpose operation, denotes the temperature parameter to be learned at the th training;

[0072] Step 6, the motion representation encoder calculates the feature mapping at the th training by using formula (4): , and between the cross-covariance attentions of the th attention head. The conventional self-attention mechanism acts on the global interaction between all tokens, which can flexibly model image features, but it has quadratic complexity and cannot be extended to high-resolution images, which is not suitable for extracting the motion representation of the input image. The proposed cross-covariance attention mechanism has linear complexity in expanding the channel dimension, which reduces the computational complexity while effectively aggregating local and global context features:

[0073] (4)

[0074] Step 7, the motion representation encoder processes the cross-covariance attentions of the n attention heads by using formula (5) to obtain the feature mapping at the th training:

[0075] (5)

[0076] In formula (5), denotes the concatenation operation of the n attention heads;

[0077] Step 8, as shown in formula (6), the motion representation encoder processes the feature mapping Figure 3 to generate the motion representation of the reference frame at the th training:

[0078] (6) In formula (6),

[0079] denotes the gating mechanism of the double-path dot product, and has:

[0080] (7) In formula (7),

[0081] denotes the expansion factor of the feature dimension, which realizes the dimensionality of the feature channel and also increases the nonlinear expression in the hidden layer; denotes the expansion factor of the feature dimension, which realizes the dimensionality of the feature channel and also increases the nonlinear expression in the hidden layer; denotes the expansion factor of the feature dimension, which realizes the dimensionality of the feature channel and also increases the nonlinear expression in the hidden layer; nonlinear activation function; , denote the Pointwise convolution operation with kernel size of and the Depthwise convolution operation with kernel size of in the first path at the i-th training, , denote the Pointwise convolution operation with kernel size of and the Depthwise convolution operation with kernel size of in the second path at the i-th training, denotes the Hadamard product; this gating mechanism of double-path dot product effectively focuses on the details of supplement and information screening, allowing useful information to enter the next layer.

[0082] Step 9, according to the process of Step 4-Step 8, and in the way of motion representation encoder sharing weights, respectively process the query feature and the perturbation feature , and correspondingly get the motion representation of the query frame and the motion representation of the color perturbed query frame . Wherein, the motion representation of the color perturbed query frame is used to calculate the motion representation loss error , see Step 15. Step 10, the motion amplification conversion module calculates the difference between the two frames of motion representation and

[0083] at the i-th training by using formula (8): (8)

[0084] (8)

[0085] Step 11, the motion amplification conversion module calculates the amplified motion representation of the difference at the i-th training by using formula (9). This module has two basic modifications to improve the learning of motion representation: (1) this module discards the convolution of large receptive field and only uses two 1x1 convolutions to interactively amplify the conversion of features across channels, which has the advantage of better compatibility with the Transformer network and contains fewer parameters; (2) using a more stable GELU activation function to provide nonlinear representation learning, thereby strengthening the dependence on features and being able to avoid gradient explosion:

[0086] ​​​​​​ (9)

[0087] Step 12, the multi-refinement decoder refines the upscaling motion representation of the first training time ; and the motion representation of the query frame ; The features are spliced along the channel dimension, thereby obtaining the upscaling fusion features of the first training time ;

[0088] Step 13, how to reconstruct the upscaling image without artifacts and less noise is very crucial. Since the latent space is low-dimensional, only the key information is extracted. The multi-refinement decoder uses a series of guided attention modules to refine the final upscaling fusion attention map and gradually eliminate the noise in the space, giving more weight to the important regions of the motion amplification. The multi-refinement decoder refines the upscaling fusion features layer by layer through the way of stacking Transformer layers to obtain fine upscaling fusion features ;

[0089] Step 14, the multi-refinement decoder reconstructs the fine upscaling fusion features using formula (10) to obtain high-quality upscaling frames of the first training time :

[0090] (10)

[0091] In formula (10), denotes the sub-pixel accuracy up-sampling operation of the first training time;

[0092] Step 15, the error of the Transformer network of the first training time is calculated using formula (11) :

[0093] (11)

[0094] In formula (11), is a hyperparameter for balancing the two loss errors, denotes the Charbonnier loss function, which has the characteristics of robustness and smoothness. Specifically, it is set to according to the experimental experience; is a constant in the Charbonnier loss function, which is ;

[0095] ​​​​​Step 16, judge whether the error of the t-th training converges; if the error converges, it indicates that the parameters of the Transformer network in the t-th training are optimal parameters Step 17, assign t+1 to t, and return to step 3 for sequential execution. Step 18, if the error of the t-th training does not converge, continue to perform gradient back propagation of the t-th training, and update the parameters of the Transformer network Step 19, after the t-th training, execute step 20. Step 20, assign t+1 to t, and return to step 3 for sequential execution. Step 21, if the error of the t-th training converges, it indicates that the parameters of the Transformer network in the t-th training are optimal parameters

[0096] Step 22, after the t-th training, execute step 23.

[0097] In this embodiment, an electronic device includes a memory for storing a program supporting the processor to execute the above method, and a processor configured to execute the program stored in the memory.

[0098] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to perform the steps of the above method.​​​​​

Claims

1. A method for video motion magnification based on a Transformer network, characterized in that The steps are as follows: Step 1, acquire continuous frame images containing micro-motion changes, and take the first frame image as a reference frame, denoted as ; take the second frame image as a query frame, denoted as ; perform color perturbation on the query frame to obtain a color-perturbed query frame, denoted as ; take the reference frame and the query frame as corresponding real magnified frames, denoted as ; take the magnification factor of the real magnified frame as , wherein represents a real number field, represents the height of an image, represents the width of an image, and 3 represents the number of channels of an image; Step 2, constructing a Transformer network, including: a motion representation encoder, a motion amplification conversion module, and a multi-refinement decoder; Let the current training number be defined as , and initialize , let the parameters of the Transformer network in the th training be ; Step 3: The motion representation encoder uses equation (1) to process the reference frames respectively. Query Frame query frames with color perturbation Feature extraction is performed, and the corresponding result is obtained. Reference features during training Query features and disturbance characteristics : (1) In formula (1), denotes the convolution operation with the kernel size of in the dimension of the feature; Step 4, the motion representation encoder processes the reference feature using equation (2) to obtain the query vector , the key vector and the value vector at the first training time : (2) In formula (2), denotes the Pointwise convolution operation at the denotes the Depthwise convolution operation at the denotes the Layer Normalization operation, denotes the Tiling operation;​​ will be divided into , and along the channel dimension of the feature respectively, so as to obtain , , and ; wherein, represents the query vector of the th attention head at the th training, represents the key vector of the th attention head at the th training, represents the value vector of the th attention head at the th training. Step 5. The motion representation encoder calculates the first second training time first transpose attention map of the attention head : (3) In formula (3), denotes an activation function, denotes a transpose operation, denotes the temperature parameter to be learned at the n-th training, training. Step 6. The motion representation encoder calculates the first training time first training time first , and intercovariance attention : (4) Step 7, the motion representation encoder processes the cross-covariance attention of the n attention heads using formula (5) to obtain the feature mapping of the first training time : (5) In formula (5), denotes concatenation operation of the attention heads. Step 8, the motion representation encoder utilizes the characteristic mapping of equation (6) processing to generate the first motion representation of the reference frame at the second training time : (6) In formula (6), denotes a gating mechanism for the two-path dot product and has: (7) In formula (7), an expansion factor representing a characteristic dimension, represents a nonlinear activation function; , respectively represent the Pointwise convolution operation and the Depthwise convolution operation under the first path at the i-th training, , , respectively represent the Pointwise convolution operation and the Depthwise convolution operation under the second path at the i-th training; , represents a Hadamard product; Step 9, process the query feature and the perturbed feature respectively according to the process of Step 4-Step 8, and in the way that the motion representation encoder shares the weight values, to obtain the motion representation of the query frame and the motion representation of the query frame with color perturbation respectively ;​ Step 10, the motion amplification conversion module calculates the first two frame motion representation and the difference : (8) Step 11, the motion amplification conversion module calculates the first difference value of the second training time using formula (9) Amplified motion representation : (9) Step 12, the multi-refinement decoder decodes the first upscaling motion representation at the second training time with the query frame motion representation feature concatenation along the channel dimension, thereby obtaining the first upscaled fusion feature at the second training time ; Step 13, the multi-refinement decoder refines the amplified fusion features layer by layer in a manner of stacking transformer layers to obtain refined amplified fusion features ;​ Step 14, the multi-refinement decoder utilizes equation (10) to the fine upsampled fused features reconstruction, obtaining the first high-quality upsampled frame : (10) In formula (10), denotes the sub-pixel accuracy upsampling operation at the n-th training Step 15, calculate the first error of the Transformer network at the second training : (11) In formula (11), is a hyperparameter balancing the two loss errors, denotes the Charbonnier loss function; Step 16: Determine the first... Error during training Whether it converges; if it converges, then it means that the first... Parameters of the Transformer network during training For optimal parameters and with optimal parameters The corresponding Transformer network serves as the final model used to generate video motion magnification; conversely, the Transformer network continues to execute the... Subgradient backpropagation is performed to update the parameters of the Transformer network. and obtained the first Parameters of the Transformer network during training Then, proceed to step 17; Step 17, assigning t+1 to t, and returning to step 3 for sequential execution.

2. An electronic device comprising a memory and a processor, characterized in that The memory is configured to store a program supporting the processor to execute the video motion amplification method of claim 1.

3. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is configured to execute the steps of the video motion amplification method of claim 1 when executed by the processor.

Images