End-to-end global and local motion estimation method based on deep learning

Through the end-to-end method of deep learning network, using feature dimensionality reduction and conditional reconstruction ideas, the problems of global and local motion estimation in the existing technology are solved, and accurate motion pattern separation and recognition in computer vision tasks are realized, which improves the Task performance.

CN116091555BActive Publication Date: 2024-12-03BEIJING UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310029285.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-01-09
Publication Date
2024-12-03
Estimated Expiration
2043-01-09

AI Technical Summary

Technical Problem

Existing motion estimation methods cannot effectively separate and estimate global and local motion in video, resulting in limited performance in specific computer vision tasks, especially because global motion methods cannot obtain local motion, and there is global and local feature space for hybrid motion estimation methods. The mutual interference of each other makes it impossible to obtain a real task-oriented motion mode.

Method used

Using an end-to-end method based on deep learning, a global and local motion estimation network is designed, and hybrid motion, global and local motion is estimated through MM module, GM module and LM module respectively, and the characteristics of dimensionality reduction and dimensionality upgrading ideas and conditional weight are used. Construct the idea, design the loss function for unsupervised learning, eliminate non-global information interference, constrain network output, and realize accurate estimation of global and local motion.

Benefits of technology

It realizes accurate estimation of global and local motion in homography and group behavior recognition tasks, improves the accuracy of deep homography and behavior recognition tasks, and effectively solves the limitations of motion estimation methods in the prior art. Improved task performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116091555B_ABST
    Figure CN116091555B_ABST
Patent Text Reader

Abstract

End-to-end global and local motion estimation methods based on deep learning belong to the field of image processing. It is very necessary to estimate global and local motion from the original video. Existing global and local motion estimation methods cannot simultaneously estimate the two motions in the video frame in an end-to-end form. The present invention proposes a three-module motion estimation network for global and local motion estimation respectively, and proposes a global motion estimator based on feature dimension transformation and global motion basis to constrain the global motion estimation module to focus on global low-rank information and exclude the interference of non-global information. The network is unsupervised deep learning using three loss functions: hybrid reconstruction loss, global reconstruction loss and local reconstruction loss. The effectiveness of the present invention is verified on the homography estimation dataset DHE and the behavior recognition dataset NCAA. Experimental results show that the present invention has better performance than previous methods.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention is based on deep learning technology, relates to the field of image processing, relates to motion pattern extraction, and specifically relates to a global and local motion pattern estimation method based on camera motion properties and local content dynamic motion properties. Background Art

[0002] Motion information in videos is used in a variety of computer vision tasks, such as behavior recognition, video stabilization, object segmentation, and target tracking. Motion information mainly arises from the relative motion of objects and cameras, based on which the motion in videos can be divided into two categories: global motion and local motion. Global motion is generated by camera motion, which can be expressed as a parameterized model, such as a homography matrix. Therefore, the degrees of freedom of global motion are limited, and its motion information should be low-rank. Unlike global motion, local motion arises from moving objects, which have arbitrary directions and distances. Therefore, local motion is often highly nonlinear and has high degrees of freedom. People have different preferences for the application of global and local motion, such as video editing algorithms mainly focus on global camera motion, while group behavior recognition algorithms focus more on local content dynamic motion. Therefore, it is necessary to obtain global and local motion separately for specific downstream applications.

[0003] As an important research topic in computer vision, current research on motion estimation can be divided into global motion estimation methods, hybrid motion estimation methods, and local motion estimation methods. These methods have their own limitations. Among them, the global motion method cannot obtain local motion and often regards it as noise. The hybrid motion estimation method has mutual interference between the global and local feature spaces and cannot obtain the real task-oriented motion pattern. In order to apply local motion to action recognition, Wu et al. published an article in 2020 titled "Fusing motion patterns and key visual information for semantic event recognition in basketball videos" and designed a method to separate local motion from hybrid motion. This method uses pre-calculated optical flow as input, and obtains the motion vectors at the four corner points by statistically analyzing the edges of the hybrid motion of each image in the video sequence. Then, a linear interpolation algorithm is used to estimate the global motion, and the local motion is obtained by subtraction. In addition, Wu et al. published an article in 2021 titled "Global Motion Estimation with Iterative Optimization-based Independent Univariate Model for Action Recognition" and proposed a local motion refinement method based on spatiotemporal thresholds. This method first uses an iterative optimization algorithm to fit global motion points to the global motion model, and then removes noise from the pre-estimated optical flow by suppressing the spatial domain threshold and optimizing the temporal domain threshold of non-global motion points to obtain the final local motion. Yang et al. published an article in 2021 titled "GLM-Net: Global and Local Motion Estimation via Task-Oriented Encoder-Decoder Structure" and obtained global and local motion by feeding the pre-computed optical flow into two encoder-decoder structures. However, all of these methods are not end-to-end because they take the pre-computed optical flow as the input of the network, and their performance is limited by the optical flow network itself, and may even fail when the optical flow is calculated incorrectly. Summary of the invention

[0004] In order to effectively solve the problems existing in existing motion estimation methods, the present invention proposes an end-to-end global and local motion estimation method based on deep learning. The present invention can take any two adjacent video frames as input and simultaneously estimate the global motion and local motion between frames.

[0005] In view of the fact that global motion itself is in a low-rank space, a global motion estimator based on the idea of ​​feature dimensionality reduction and dimensionality increase is designed to extract the global information in the mixed motion and eliminate the interference of non-global information.

[0006] Aiming at the characteristic that mixed motion contains global and local motion, a training strategy based on conditional reconstruction is designed. Three loss functions are used to constrain the output of the network, so that the network can learn the mixed, global and local motion information in two adjacent video frames in an unsupervised manner.

[0007] The global motion and local motion estimated by the present invention have been well applied in homography estimation tasks and group behavior recognition tasks, and have a good performance improvement effect on subsequent tasks. The overall framework of the algorithm of the invention is as follows Figure 1 shown.

[0008] The specific steps of the present invention are as follows:

[0009] 1) Build an end-to-end deep learning network for global and local motion estimation.

[0010] The motion information in the video includes global motion and local motion, which have different sources, such as the camera and independently moving objects. In specific applications, the two are often superimposed in the form of optical flow to form a dense motion field that reflects the relative motion between frames. However, only a single type of global motion or local motion is required to complete different tasks: for example, in group behavior recognition tasks, local motion can directly express the motion category, while in video stabilization tasks, global motion is more helpful. Therefore, it is necessary to estimate the global and local motion in mixed motion separately from most video tasks.

[0011] Mixed motion is the motion of all contents in the video, presented in the form of optical flow. In order to estimate global and local motions simultaneously, the present invention adopts three modules to implement this network, namely, MM module, GM module and LM module, so as to estimate mixed motion, global motion and local motion. Since the information of global motion and local motion is mixed in the image content of the video frame, it is challenging to directly estimate global or local motion. Therefore, the present invention first estimates mixed motion based on two adjacent video frames input, and then estimates global and local motion respectively according to the mixed information in the mixed motion.

[0012] Specifically, two adjacent video frames are input into three deep learning network structures, namely, MM module, GM module and LM module, to obtain mixed, global and local motions. For the MM module, two adjacent video frames I1 and I2 are input into the pyramid feature extractor of the PWC-Net network to obtain multi-scale feature pairs. The feature pyramid extractor consists of 6 layers of convolution, and the number of input and output channels is 3->16, 16->32, 32->64, 64->96, 96->128, 128->196, respectively, with a step size of 2 and a convolution kernel size of 3×3, so that the input image is downsampled 6 times. Except for the last layer of convolution, each layer of convolution has a LeakyReLU activation layer. Then their cost volume is calculated to measure the correlation between different features and input to the optical flow decoder to obtain a mixed optical flow. The input of the optical flow decoder is the cost volume of the current layer, the optical flow obtained by doubling the optical flow output of the previous layer (the value is 0 when it is the top layer), and the features output by the feature pyramid extractor with the first frame I1 as input. The features of the first frame I1 image are subjected to a 1×1 convolution to change the number of channels from 192 to 32. The convolution kernel size of the 1×1 convolution is 1×1, and the step size is 1. The number of input and output channels of the convolution layer of the optical flow decoder is 113->128, 128->128, 256->96 (256 is the number of channels of the new feature obtained by connecting the features of the first layer output to the features of the second layer output), 224->64 (224 is the number of channels of the new feature obtained by connecting the features of the second layer output to the features of the third layer output), 160->32 (160 is the number of channels of the new feature obtained by connecting the features of the third layer output to the features of the fourth layer output), 96->2 (96 is the number of channels of the new feature obtained by connecting the features of the fourth layer output to the features of the fifth layer output), the stride is 2, and the convolution kernel is 3×3. Except for the last layer, each convolution layer has a LeakyReLU activation layer. The output of the MM module is the optical flow of the current layer, which is then input to the GM module and the LM module to obtain global and local motion.

[0013] Inspired by the unsupervised optical flow estimation method, the present invention designs a loss function based on the conditional reconstruction idea to solve the problem faced by the current global and local motion estimation methods, which is the lack of effective supervision due to the lack of labeled data sets. Furthermore, considering that global motion is a low-degree-of-freedom motion with limited motion types, the present invention constrains the global motion to a low-rank subspace, thereby obtaining a less noisy global motion estimation result that excludes the local motion area. Given that the method of the present invention can learn superimposed motion information, once a more accurate global motion is obtained, a more accurate local motion can be obtained.

[0014] 2) Construct a global motion estimator based on the idea of ​​feature dimensionality reduction and dimensionality increase to obtain low-rank global motion estimation results.

[0015] Global motion comes from the self-motion of the camera, while camera motion does not produce deformation and is a rigid motion. For camera motion estimation, existing methods use a specific 8-parameter quadratic motion model (such as a homography matrix) to correspond to the projection of this rigid motion of the 3D plane in the image plane. Therefore, the camera motion is passed through a mask and a linear dimensionality reduction operation, and the global motion can be represented by a low-rank motion vector. The present invention uses a global motion estimator to filter out non-global information in the feature space. The mixed motion is encoded into a low-dimensional feature vector and then decoded into a high-dimensional feature vector. The high-dimensional feature vector is further constrained to be a linear combination of 8 low-rank and orthogonal motion basis vectors.

[0016] The definition of motion basis comes from the article "Unsupervised Global and Local Homography Estimation with Motion Basis Learning" published by Liu et al. in 2022. Since the motion basis vector itself is low-rank and the global motion is a linear combination of it, the result eliminates the visual interference of local motion, thereby obtaining a cleaner global motion estimation effect. Based on this, it can be guaranteed that all information in the global motion is low-rank and can be used to express global camera motion.

[0017] First, use the submodule of the global motion estimator Filter out non-global information in the feature space. Encode the mixed motion into a one-dimensional vector F of length 8 l , and then decoded into a one-dimensional vector F of length 65536 h The feature vector F is further transformed using the submodule ε of the global motion estimator. h The constraint is a weight vector of length 8, which represents a linear combination of 8 low-rank and orthogonal motion basis vectors. The motion basis is obtained by QR decomposition of the homography matrix, which represents the 8 basic transformations of camera motion. The operation process of the global motion estimator is as follows:

[0018]

[0019] where U is the blended motion generated by the MM module, which is set to a one-dimensional vector of length 65536, It is a submodule of the global motion estimator The encoder consists of three layers of perceptrons, and the dimensions of the input and output vectors are 65536->1024, 1024->128, and 128->8 respectively. is with Symmetrical The decoder consists of three layers of perceptrons, with input and output vector dimensions of 8->128, 128->1024, and 1024->65536. Except for the last layer, batch normalization is used after each layer of the multi-layer perceptron to improve the gradient conduction of the network. At the same time, the hyperbolic tangent function (Tanh) is used as the nonlinear activation function after each batch normalization to control the output range to (-1,1) and retain the motion information in the negative value. F h It is a decoder The output is a one-dimensional vector, F h =65536.

[0020] {ω i}=ε(F h ), (2)

[0021]

[0022] Among them, {ω i} is the weight set of motion basis vectors, which is a one-dimensional vector of length 8. ε is a submodule of the global motion estimator that contains 2 convolutional layers and 1 adaptive average pooling layer. The first convolutional layer of ε has 2 input channels, 512 output channels, a convolution kernel size of 7×7, a stride of 2, and a padding of 3. The second convolutional layer is a grouped convolution with 8 groups, 512 input channels, 8 output channels, a convolution kernel size of 1×1, a stride of 1, and a padding of 0. g i is the motion basis vector with index i, N is the number of motion bases, N=8. It is the domain where the motion basis vector is located, which is the motion image domain of number of channels × height × width = 2 × H × W, H = 128, W = 256. Since the motion basis used by the global motion estimator expresses the motion of the camera, using its linear combination to express the global motion can ensure that all the information in the global motion is low-rank.

[0023] 3) Construct a deep learning network training strategy based on the idea of ​​conditional reconstruction to constrain the network output so that the network can learn the mixed, global and local motion information in two adjacent video frames in an unsupervised manner.

[0024] The idea of ​​conditional reconstruction comes from the assumption of photometric constancy, which is a classic assumption for unsupervised optical flow estimation. The present invention adopts a hybrid reconstruction loss based on photometric loss and smoothness loss for the MM module, so as to learn mixed motion from two adjacent video frames in an unsupervised manner. The photometric loss measures the error between the first frame image I1 and the reconstructed first frame I′1, where the reconstructed first frame I′1 is obtained by warping the second frame image I2. The hybrid reconstruction loss is defined as follows:

[0025]

[0026] Where S is the structural similarity (SSIM) error. For the warping operation, let p be the pixel coordinate on the image plane, U be the estimated mixed motion, and U(p) represent the motion trajectory of the pixel point with coordinate p. Then the pixel value I′1(p) at the pixel coordinate p of the reconstructed first frame I′1 satisfies I′1(p)=I2(p+U(p)).

[0027] The smoothing loss is defined as follows:

[0028]

[0029] in is the gradient of the mixed motion U in the x-axis direction, is the gradient of the mixed motion U in the y-axis direction, is the gradient of the first frame image I1 in the x-axis direction, is the gradient of the first frame image I1 in the y-axis direction, ‖‖1 is the L1 norm operation, || means taking the absolute value, and exp() means the power of the natural constant e. The final hybrid reconstruction loss is defined as follows:

[0030]

[0031] where λ ph and λ sm are weight terms, which are 1 and 0.5 respectively.

[0032] The global reconstruction loss of the GM module measures the error between the first frame I1 and the reconstructed first frame obtained by the warping operation of the second frame I2. The details are as follows:

[0033]

[0034] Among them, p is a pixel on the image plane, ‖‖2 is the L2 norm operation, G is the global motion estimated by the GM module, and B g It is a binary mask generated based on an iterative optimization algorithm. The iterative optimization algorithm uses the least squares method to fit the function of global motion to the optical flow and estimate the model parameters. Points that cannot fit the function are set as outliers and their values ​​are set to 0 to remove pixels with foreground motion. The remaining values ​​are set to 1 to retain the remaining background area.

[0035] The local reconstruction loss of the LM module measures the accuracy of local motion after removing scene-irrelevant areas. The local reconstruction loss function of the LM module is as follows:

[0036]

[0037] Where p is a pixel on the image plane, U is the mixed motion estimated by the MM module, G is the global motion estimated by the GM module, L is the local motion estimated by the LM module, ‖‖2 is the L2 norm operation, and B r is a manually annotated mask used to remove scene irrelevant areas. r The pixel value of the area is 1. By training to reduce the error between mixed motion and global plus local motion, the network can be constrained to learn task-driven local motion.

[0038] The final conditional reconstruction loss function is as follows:

[0039]

[0040] where λ g and λ l is the weight term. When the training epoch is between 1 and 50, it takes 1 and 0.0001. When the epoch is between 51 and 100, it takes 1 and 0.002. When the epoch is between 101 and 200, it takes 1 and 1. The total number of training epochs is 200.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] The present invention proposes an unsupervised learning framework, which can estimate global motion and local motion simultaneously in an end-to-end manner. The present invention designs a global motion estimator to estimate global motion. The estimator can constrain the GM module to focus on global low-rank information for global motion estimation, and generate global motion that eliminates irrelevant information. In order to achieve the learning of the network, the present invention adopts three loss functions of hybrid motion estimation, global motion estimation and local motion estimation based on the idea of ​​conditional reconstruction, and performs unsupervised estimation of global and local motion in an end-to-end manner, effectively improving the accuracy of deep homography estimation and behavior recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0044] The present invention proposes an end-to-end global and local motion estimation method based on deep learning. The specific implementation steps of the invention are as follows:

[0045] Step 1: The present invention specifically includes three main modules, namely, MM module, GM module and LM module. Among them, the MM module is responsible for converting two adjacent video frames into mixed motion (optical flow), and based on the assumption of photometric constancy, minimizes the loss function between the frame reconstructed by the warping operation and the original frame, thereby obtaining a mixed global and local motion. The MM module adopts the PWC-Net structure, which specifically includes three main components: a feature pyramid extractor, a cost body layer and an optical flow decoder. The input two-frame sequence is fed to the feature pyramid extractor to extract pyramid feature pairs, and then the pyramid feature pairs and their cost bodies are estimated by the optical flow decoder in a coarse-to-fine manner. Optical flow, so the output of this module is a dense motion field mixed with global and local motion.

[0046] Specifically, multi-scale feature pairs are obtained by inputting video frames I1 and I2 into the pyramid feature extractor of the PWC-Net network. The feature pyramid extractor consists of 6 layers of convolution, and the number of input and output channels is 3->16, 16->32, 32->64, 64->96, 96->128, 128->196, respectively. The step size is 2, and the convolution kernel size is 3×3, so that the input image is downsampled 6 times. Except for the last layer of convolution, each layer of convolution has a LeakyReLU activation layer. Then their cost volume (CostVolume) is calculated to measure the correlation between different features, and input to the optical flow decoder to obtain a mixed optical flow. The input of the optical flow decoder is the cost volume of the current layer, the optical flow obtained by doubling the optical flow output of the previous layer (the value is 0 when it is the top layer), and the features output by the feature pyramid extractor with the first frame I1 as input. The features of the first frame I1 image are subjected to a 1×1 convolution to change the number of channels from 192 to 32. The convolution kernel size of the 1×1 convolution is 1×1, and the step size is 1. The number of input and output channels of the convolution layer of the optical flow decoder is 113->128, 128->128, 256->96 (256 is the number of channels of the new feature obtained by connecting the features of the first layer output to the features of the second layer output), 224->64 (224 is the number of channels of the new feature obtained by connecting the features of the second layer output to the features of the third layer output), 160->32 (160 is the number of channels of the new feature obtained by connecting the features of the third layer output to the features of the fourth layer output), 96->2 (96 is the number of channels of the new feature obtained by connecting the features of the fourth layer output to the features of the fifth layer output), the stride is 2, and the convolution kernel is 3×3. Except for the last layer, each convolution layer has a LeakyReLU activation layer. The output of the MM module is the optical flow of the current layer, which is then input to the GM module and the LM module to obtain global and local motion. The GM module compresses, restores and filters out non-global information on the mixed motion to obtain the global motion. The LM module performs task-oriented local motion estimation on the estimated mixed motion and global motion.

[0047] Step 2: In view of the fact that global motion is caused by camera motion such as translation and shaking, the present invention can be modeled by a geometric transformation model with low degrees of freedom, so the global motion should be low-rank. If the global motion is expressed by an 8-degree-of-freedom homography matrix model, the rank of the global motion should be 8. Accordingly, estimating the global motion only requires extracting the low-rank information representing the global motion in the mixed motion, and in this process, the high-rank information such as local motion is regarded as noise and eliminated. Therefore, the present invention proposes to filter out non-global information at the feature level by compressing the mixed motion output by the MM module into a low-dimensional feature vector and restoring it to a high-dimensional feature. The global motion is further expressed as a linear combination of 8 low-rank and orthogonal motion basis vectors, thereby expressing the global motion as a low-rank optical flow field, which naturally excludes the existence of local motion. Specifically, first use the submodule of the global motion estimator Filter out non-global information in the feature space. Encode the mixed motion into a one-dimensional vector F of length 8 l , and then decoded into a one-dimensional vector F of length 65536 h The feature vector F is further transformed using the submodule ε of the global motion estimator. h The constraint is a weight vector of length 8, which represents a linear combination of 8 low-rank and orthogonal motion basis vectors. The motion basis is obtained by QR decomposition of the homography matrix, which represents the 8 basic transformations of camera motion. The operation process of the global motion estimator is as follows:

[0048]

[0049] where U is the blended motion generated by the MM module, which is set to a one-dimensional vector of length 65536, It is a submodule of the global motion estimator The encoder consists of three layers of perceptrons, and the dimensions of the input and output vectors are 65536->1024, 1024->128, and 128->8 respectively. is with Symmetrical The decoder consists of three layers of perceptrons, with input and output vector dimensions of 8->128, 128->1024, and 1024->65536. Except for the last layer, batch normalization is used after each layer of the multi-layer perceptron to improve the gradient conduction of the network. At the same time, the hyperbolic tangent function (Tanh) is used as the nonlinear activation function after each batch normalization to control the output range to (-1,1) and retain the motion information in the negative value. F h It is a decoder The output is a one-dimensional vector, F h =65536.

[0050] {ω i}=ε(F g ), (2)

[0051]

[0052] Among them, {ω i} is the weight set of motion basis vectors, which is a one-dimensional vector of length 8. ε is a submodule of the global motion estimator that contains 2 convolutional layers and 1 adaptive average pooling layer. The first convolutional layer of ε has 2 input channels, 512 output channels, a convolution kernel size of 7×7, a stride of 2, and a padding of 3. The second convolutional layer is a grouped convolution with 8 groups, 512 input channels, 8 output channels, a convolution kernel size of 1×1, a stride of 1, and a padding of 0. g i is the motion basis vector with index i, N is the number of motion bases, N=8. It is the domain where the motion basis vector is located, which is the motion image domain of number of channels × height × width = 2 × H × W, H = 128, W = 256. Since the motion basis used by the global motion estimator expresses the motion of the camera, using its linear combination to express the global motion can ensure that all the information in the global motion is low-rank.

[0053] Step 3: The present invention adopts three loss functions, namely, hybrid reconstruction loss, global reconstruction loss and local reconstruction loss, corresponding to MM, GM and LM modules respectively. The hybrid reconstruction loss is designed based on unsupervised optical flow loss, and the hybrid motion is estimated by calculating the point-by-point error between the original video frame and the reconstructed video frame, and introducing the local smoothness assumption. When calculating the global reconstruction loss, a non-global mask of the hybrid motion is generated by an iterative optimization algorithm to guide the loss function based on the assumption of constant luminosity. When calculating the local reconstruction loss, the present invention obtains the reconstructed hybrid motion by mixing the estimated global and local motions, further eliminates the interference of the scene-irrelevant area by manually setting the mask, and calculates the point-by-point error of the remaining area between the reconstructed hybrid motion and the hybrid motion estimated by the MM module. The network training strategy is as follows: the height of the two adjacent video frames input is 128, the width is 256, the number of channels is 3, the Adam optimizer update strategy is adopted during network training, the training batch size is 64, the initial learning rate is 0.001, and the training round is 200 rounds.

[0054] Specifically, a hybrid reconstruction loss based on photometric loss and smoothness loss is used for the MM module to learn mixed motion from two adjacent video frames in an unsupervised manner. The photometric loss measures the difference between the first frame image I1 and the reconstructed first frame I1. ′ The error between the first reconstructed frame I1 ′The second frame image I2 is obtained by warping. The hybrid reconstruction loss is defined as follows:

[0055]

[0056] Where S is the structural similarity (SSIM) error. For the warping operation, let p be the pixel coordinates on the image plane, U be the estimated mixed motion, and U(p) represent the motion trajectory of the pixel point with coordinate p. Then the reconstructed first frame I1 ′ The pixel value I1 at pixel coordinate p ′ (p) Satisfy I1 ′ (p) = I2(p+U(p)).

[0057] The smoothing loss is defined as follows:

[0058]

[0059] in is the gradient of the mixed motion U in the x-axis direction, is the gradient of the mixed motion U in the y-axis direction, is the gradient of the first frame image I1 in the x-axis direction, is the gradient of the first frame image I1 in the y-axis direction, ‖‖1 is the L1 norm operation, || means taking the absolute value, and exp() means the power of the natural constant e. The final hybrid reconstruction loss is defined as follows:

[0060]

[0061] where λ ph and λ sm are weight terms, which are 1 and 0.5 respectively.

[0062] The global reconstruction loss of the GM module measures the error between the first frame I1 and the reconstructed first frame obtained by the warping operation of the second frame I2. The details are as follows:

[0063]

[0064] Among them, p is a pixel on the image plane, ‖‖2 is the L2 norm operation, G is the global motion estimated by the GM module, and B g It is a binary mask generated based on an iterative optimization algorithm. The iterative optimization algorithm uses the least squares method to fit the function of global motion to the optical flow and estimate the model parameters. Points that cannot fit the function are set as outliers and their values ​​are set to 0 to remove pixels with foreground motion. The remaining values ​​are set to 1 to retain the remaining background area.

[0065] The loss function of the LM module is as follows:

[0066]

[0067] Where p is a pixel on the image plane, U is the mixed motion estimated by the MM module, G is the global motion estimated by the GM module, L is the local motion estimated by the LM module, ‖‖2 is the L2 norm operation, and B r is a manually annotated mask used to remove scene irrelevant areas. r The pixel value of the area is 1. By training to reduce the error between mixed motion and global plus local motion, the network can be constrained to learn task-driven local motion.

[0068] The final conditional reconstruction loss function is as follows:

[0069]

[0070] where λ g and λ l is the weight term. When the training epoch is between 1 and 50, it takes 1 and 0.0001. When the epoch is between 51 and 100, it takes 1 and 0.002. When the epoch is between 101 and 200, it takes 1 and 1. The total number of training epochs is 200.

[0071] In order to verify the effectiveness of the present invention, the global motion and local motion estimation results of the algorithm of the present invention were experimented on the homography estimation dataset DHE and the behavior recognition dataset NCAA. First, the performance of the global motion of the present invention on the homography estimation task was tested. As shown in Table 1, the present invention achieved comparable performance compared with the current best method, namely the content-aware method "Content-Aware Unsupervised Deep Homography Estimation" proposed by Liu Shuaicheng's team and the encoding and decoding method "GLM-Net: Global and Local Motion Estimation via Task-Oriented Encoder-Decoder Structure" proposed by Wu Lifang's team, and obtained better results in the conventional (RE) scenario.

[0072] Table 1: Performance comparison of homography estimation tasks

[0073] RE LL SF LF Content-aware 1.81 1.94 1.75 1.72 Encoding and Decoding 1.81 1.95 1.97 2.07 Patented method 1.76 2.04 2.22 2.25

[0074] In order to compare with the best method at present, namely the encoding and decoding method "GLM-Net: Global and Local Motion Estimation via Task-Oriented Encoder-Decoder Structure" proposed by Wu Lifang's team, and related methods: statistical analysis method "Fusing motion patterns and key visual information for semantic event recognition in basketball videos" and iterative optimization method "Global motion estimation with iterative optimization-based independent univariate model for action recognition", the present invention adopts the same settings as these methods, and compares the global motion area estimated by different methods with the corresponding area of ​​optical flow point by point, so as to compare the performance of global motion estimated by different methods. The results are shown in Table 2. Compared with these methods, the present invention has the smallest error and achieves the best global motion estimation effect, which proves the effectiveness of the global motion estimation method in the present invention.

[0075] Table 2: Performance comparison of global motion estimation

[0076]

[0077]

[0078] Furthermore, the present invention refers to the encoding and decoding method, performs group behavior recognition tasks on the NCAA data set, and compares with the best methods currently available. The present invention uses the C3D network to classify local motion in mixed motion, and the results are shown in Table 3. Compared with other methods, namely the iterative optimization method and encoding and decoding method proposed by Wu Lifang's team, the estimated local motion of the present invention is more conducive to the behavior recognition task, and the average accuracy of the event classification obtained is higher, which proves the effectiveness of the local motion obtained by the present invention.

[0079] Table 3: Group behavior recognition accuracy on the NCAA dataset

[0080] Statistical analysis Iterative Optimization Encoding and Decoding Patented method 3-pointer 0.695 0.802 0.777 0.857 penalty 0.821 0.866 0.910 0.851 Layup 0.626 0.593 0.594 0.463 2-point shot 0.563 0.582 0.614 0.568 Dunk 0.389 0.333 0.278 0.611 Steals 0.940 0.976 0.993 0.942 average 0.672 0.688 0.690 0.734

Claims

1. A deep learning based end-to-end global and local motion estimation method, characterized by: An end-to-end deep learning network for estimating global and local motion is designed, which estimates the global and local motion of two adjacent video frames as input; Considering the fact that global motion is in a low-rank space, a global motion estimator based on the idea of ​​feature dimensionality reduction and dimensionality increase is designed to extract global information from mixed motion and eliminate the interference of non-global information. Aiming at the characteristic that mixed motion contains global and local motion, a training strategy based on conditional reconstruction is designed. Three loss functions are used to constrain the output of the network, so that the network can learn the mixed, global and local motion information in two adjacent video frames in an unsupervised manner. In step (1), global and local motions are obtained by inputting two adjacent video frames into three deep learning network structures, namely, MM module, GM module and LM module; for the MM module, multi-scale feature pairs are obtained by inputting video frames I1 and I2 into the pyramid feature extractor of the PWC-Net network. The feature pyramid extractor consists of 6 layers of convolution, and the number of input and output channels is 3->16, 16->32, 32->64, 64->96, 96->128, 128->196, respectively. The step size is 2 and the convolution kernel size is 3×3, so that the input image is downsampled 6 times; except for the last layer of convolution, each layer of convolution has a LeakyReLU activation layer; then their cost bodies are calculated to measure the correlation between different features, and are input to the optical flow decoder to obtain a mixed optical flow; the input of the optical flow decoder is the cost body of the current layer, the optical flow obtained by doubling the optical flow output of the previous layer, and the features output by the feature pyramid extractor input with the first frame I1. The features of the first frame I1 image in the image are transformed from 192 to 32 through 1×1 convolution. The convolution kernel size of 1×1 convolution is 1×1, and the step size is 1. The input and output channels of the convolution layer of the optical flow decoder are 113->128, 128->128, 256->96, where 256 is the number of channels of the new features obtained by connecting the features output by the first layer to the features output by the second layer; 224->64 is the number of channels of the new features obtained by connecting the features output by the second layer to the features output by the third layer. The number of channels is 160->32, where 160 is the number of channels of the new feature obtained by connecting the features output by the third layer with the features output by the fourth layer; 96->2, where 96 is the number of channels of the new feature obtained by connecting the features output by the fourth layer with the features output by the fifth layer; the step size is 2, and the convolution kernel is 3×3; except for the last layer, each convolution layer has a LeakyReLU activation layer; the output of the MM module is the optical flow of the current layer, which is then input to the GM module and the LM module to obtain global and local motion; In step (2), given that the global motion originates from the camera motion with low degrees of freedom, it can be expressed as low-rank motion information; first, a submodule of the global motion estimator is used Filter out non-global information in the feature space; encode the mixed motion into a length of F l =8, and then decoded into a one-dimensional vector of length F h = a one-dimensional vector of 65536; use the submodule ε of the global motion estimator to convert the feature vector F h The constraint is a weight vector of length 8, which represents the linear combination of 8 low-rank and orthogonal motion basis vectors; the motion basis is obtained by QR decomposition of the homography matrix, which represents the 8 basic transformations of camera motion; The operation process of the global motion estimator is as follows: Where U is the hybrid motion generated by the MM module, which is set to be of length K m = a one-dimensional vector of 65536, It is a submodule of the global motion estimator The encoder consists of three layers of perceptrons, and the dimensions of the input and output vectors are 65536->1024, 1024->128, and 128->8 respectively; is with Symmetrical The decoder consists of three layers of perceptrons, with input and output vector dimensions of 8->128, 128->1024, and 1024->65536 respectively. Except for the last layer, batch normalization is used after each layer of the multilayer perceptron to improve the gradient conduction of the network. At the same time, the hyperbolic tangent function is used as the nonlinear activation function after each batch normalization to control the output range to (-1,1) and retain the motion information in the negative value. h It is a decoder The output is a one-dimensional vector, F h =65536; {oh} i }=ε(F h ), (2) Among them, {ω i } is the weight set of motion basis vectors, which is a one-dimensional vector of length 8. ε is a submodule of the global motion estimator containing 2 convolutional layers and 1 adaptive average pooling layer. The first convolutional layer of ε has 2 input channels, 512 output channels, 7×7 convolution kernel size, 2 stride, and 3 padding. The second convolutional layer is a grouped convolution with 8 groups, 512 input channels, 8 output channels, 1×1 convolution kernel size, 1 stride, and 0 padding. g i is the motion basis vector with index i, N is the number of motion bases, N = 8; is the domain of the motion basis vector, which is the motion image domain of number of channels × height × width = 2 × H × W, H = 128, W = 256; In step (3), a hybrid reconstruction loss based on photometric loss and smoothness loss is used for the MM module to learn mixed motion from two adjacent video frames in an unsupervised manner; the photometric loss measures the error between the first frame image I1 and the reconstructed first frame I′1, where the reconstructed first frame I′1 is obtained by warping the second frame image I2; the hybrid reconstruction loss is defined as follows: Where S is the structural similarity (SSIM) error; for the warping operation, let p be the pixel coordinate on the image plane, U be the estimated mixed motion, and U(p) represent the motion trajectory of the pixel point with coordinate p, then the reconstructed first frame I1 ′ The pixel value I1 at pixel coordinate p ′ (p) Satisfy I1 ′ (p) = I2(p+U(p)); The smoothing loss is defined as follows: in is the gradient of the mixed motion U in the x-axis direction, is the gradient of the mixed motion U in the y-axis direction, is the gradient of the first frame image I1 in the x-axis direction, is the gradient of the first frame image I1 in the y-axis direction, ‖‖1 is the L1 norm operation, || means taking the absolute value, and exp() means the power of the natural constant e; the final hybrid reconstruction loss is defined as follows: where λ ph and λ sm are weight items, which are 1 and 0.5 respectively; The global reconstruction loss of the GM module measures the error between the first frame I1 and the reconstructed first frame obtained by the warping operation of the second frame I2; specifically, it is as follows: Among them, p is a pixel on the image plane, ‖‖2 is the L2 norm operation, G is the global motion estimated by the GM module, and B g It is a binary mask generated based on an iterative optimization algorithm. The iterative optimization algorithm uses the least squares method to fit the function of global motion to the optical flow and estimates the model parameters. The points that cannot fit the function are set as outliers and their values ​​are set to 0 to remove pixels with foreground motion. The remaining values ​​are set to 1 to retain the remaining background area. The loss function of the LM module is as follows: Where p is a pixel on the image plane, U is the mixed motion estimated by the MM module, G is the global motion estimated by the GM module, L is the local motion estimated by the LM module, ‖‖2 is the L2 norm operation, and B r is a manually annotated mask used to remove scene-irrelevant areas; in mask B r The pixel value of the area is 1. By training, the error between mixed motion and global plus local motion can be reduced, and the network can be constrained to learn task-driven local motion. The final conditional reconstruction loss function is as follows: where λ g and λ l is the weight term. When the training epoch is between 1 and 50, it takes 1 and 0.0001. When the epoch is between 51 and 100, it takes 1 and 0.

002. When the epoch is between 101 and 200, it takes 1 and 1. The total number of training epochs is 200.