Video stream inter-frame motion parameter estimation method, video stabilization method and product

Abstract

The invention provides a video stream inter-frame motion parameter estimation method, a video stabilization method and a product. The method comprises the following steps: acquiring a current frame in real time; predicting to obtain predicted inter-frame motion parameters of the current frame relative to the previous frame by using a neural network; obtaining a corrected intermediate frame based on the predicted inter-frame motion parameter; performing feature point detection on the grey-scale map of the previous frame to obtain a target feature point, calculating the relative movement of the target feature point between the current frame and the previous frame by using an optical flow method, and selecting an effective feature point pair; and determining an accurate affine transformation matrix of the current frame relative to the previous frame, extracting an actual inter-frame motion parameter, and according to the difference between the actual inter-frame motion parameter and the predicted inter-frame motion parameter, carrying out online updating on the parameters of the neural network model by using the difference. According to the technical scheme, a rough prediction and accurate estimation dual-stage motion parameter estimation method is introduced, so that the estimation precision is ensured while the estimation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video de-shaking, and in particular to a method for estimating motion parameters between video stream frames, a video stabilization method and a computer program product. Background Art

[0002] At present, video capture is widely used in various scenarios, and its functions are also various, such as entertainment and monitoring. For example, in industrial ports, cameras are used for video capture. Some of these cameras are installed on port equipment. Due to the complexity of the actual port environment, various jitters that may be generated by equipment such as large equipment, and the movement of equipment installed with cameras, the video and image stabilization captured by cameras in industrial ports face many challenges. In particular, the complexity of the environment makes the estimation of the jitter parameters of the existing video frames, that is, the inter-frame motion parameters, have the problem of insufficient estimation accuracy and low estimation efficiency during the video stabilization or de-jittering process. Summary of the invention

[0003] Embodiments of the present invention provide a method for estimating inter-frame motion parameters of a video stream, a video stabilization method, and a computer program product, which estimate inter-frame motion parameters by combining the dual stages of prediction transformation and precise optical flow, thereby improving estimation efficiency while ensuring estimation accuracy.

[0004] In order to achieve the above object, on the one hand, a method for estimating motion parameters between frames of a video stream is provided, comprising:

[0005] Get the current frame in real time;

[0006] According to the actual inter-frame motion parameters between each frame in the N frames of images before the current frame and the previous frame adjacent to the current frame, the translation trajectory of the N frames of images in a predetermined x-axis direction, the translation trajectory of the N frames of images in a predetermined y-axis direction, the rotation trajectory of the N frames of images and the scaling trajectory of the N frames of images, a predicted inter-frame motion parameter of the current frame relative to the previous frame adjacent to the current frame is predicted using a pre-trained neural network model, wherein the inter-frame motion parameters include: a translation motion parameter Δx in a predetermined x-axis direction, a translation motion parameter Δy in a predetermined y-axis direction, a rotation angle Δθ and a scaling factor Δs;

[0007] Generate a first affine transformation matrix of the current frame relative to a previous frame adjacent to the current frame based on the predicted inter-frame motion parameter;

[0008] Apply the first affine transformation matrix to the current frame to obtain a corrected intermediate frame, where I warp =M predict *I current , I currentis the current frame, I warp is the corrected intermediate frame, M predict is the first affine transformation matrix;

[0009] Using a predetermined feature point detection algorithm to perform feature point detection on a grayscale image of a previous frame adjacent to the current frame to obtain a plurality of target feature points, and using a selected optical flow method to calculate the relative motion of the plurality of target feature points between the current frame and the previous frame adjacent to the current frame;

[0010] According to the relative motion of the plurality of target feature points between the current frame and a previous frame adjacent to the current frame, a feature point pair satisfying a predetermined condition is selected as a valid feature point pair, wherein the predetermined condition includes: a predetermined motion consistency condition and / or a preset stability condition;

[0011] Use the selected valid feature point pairs to obtain a second affine transformation matrix M of the intermediate frame relative to the previous frame adjacent to the current frame. correction , and according to the second affine transformation matrix M correction Determine a third affine transformation matrix M of the current frame relative to a previous frame adjacent to the current frame accurate ,in:

[0012] I previous =M correction *I warp =M correction *M predict *I current ; Among them, I previous is a previous frame adjacent to the current frame;

[0013] M accurate =M correction *M predict ;

[0014] I previous =M accurate *I current ;

[0015] From the third affine transformation matrix M accurate The actual inter-frame motion parameters of the current frame relative to a previous frame adjacent to the current frame are extracted, the difference between the actual inter-frame motion parameters and the predicted inter-frame motion parameters is obtained, the difference between the actual inter-frame motion parameters and the predicted inter-frame motion parameters is stored, and when the number of the stored differences reaches a first predetermined number, the parameters of the neural network model are updated online.

[0016] Preferably, the method for estimating inter-frame motion parameters of a video stream, wherein the step of predicting the predicted inter-frame motion parameters of the current frame relative to the previous frame adjacent to the current frame using a pre-trained neural network model based on the pre-obtained actual inter-frame motion parameters between each frame in the N frames of images before the current frame and the previous frame adjacent to itself: the input parameters of the neural network model include: the pre-stored actual inter-frame motion parameters between each frame in the N frames of images and the previous frame adjacent to itself; the translation trajectory of the N frames of images in the x-axis direction, the translation trajectory of the N frames of images on the y-axis, the rotation trajectory of the N frames of images and the scaling trajectory of the N frames of images, wherein the translation trajectory, the rotation trajectory and the scaling trajectory are determined by the following formula:

[0017] x(t)=x(t-1)+Δx(t), x(0)=0,

[0018] y(t)=y(t-1)+Δy(t), y(0)=0,

[0019] θ(t-1)+Δθ(t), θ(0)=0,

[0020] s(t)=s(t-1)×Δs(t), s(0)=1,

[0021] Where x(t) is the position in the x-axis direction at time t; y(t) is the position in the y-axis direction at time t; θ(t) is the rotation angle at time t; s(t) is the scaling factor at time t; Δx(t) is the translation in the x-axis direction at time t; Δy(t) is the translation in the y-axis direction at time t; Δθ(t) is the change in the rotation angle at time t, and Δs(t) is the change in the scaling factor at time t.

[0022] The output of the neural network model includes: the predicted inter-frame motion parameters of the current frame relative to the previous frame adjacent to the current frame; and the confidence probability of this prediction.

[0023] Preferably, the method for estimating inter-frame motion parameters of a video stream, wherein the neural network model includes an input layer, a first LSTM layer, a second LSTM layer, a dense layer and an output layer connected in sequence, and the output layer includes a mean output layer and a confidence output layer; wherein the input layer is used to receive the input parameters of a predetermined time step; the first LSTM layer is used to process the data output by the input layer and return the output of all time steps; the second LSTM layer is used to process the data output by the first LSTM layer and only return the output of the last time step; the dense layer is used to process the data output by the second LSTM layer using a nonlinear transformation; the mean output layer is used to process the data output by the dense layer and output the mean of each motion parameter in the predicted inter-frame motion parameters; the confidence output layer is used to process the data output by the dense layer and output the confidence probability corresponding to this prediction.

[0024] Preferably, in the method for estimating inter-frame motion parameters of a video stream, the step of training the neural network model comprises: for each of the N training samples, calculating the mean square error between the predicted inter-frame motion parameters predicted by the neural network model and the actual inter-frame motion parameters, and using the mean square error as the prediction error, wherein:

[0025] Where ΔX pred,i , Δy pred,i , Δθ pred,i , Δs pred,i is the predicted inter-frame motion parameter of the i-th training sample, Δx true,i , Δy tru,i , Δθ truei , Δs true,i is the actual inter-frame motion parameter of the i-th training sample, MSE i is the mean square error of the i-th training sample;

[0026] A confidence label corresponding to the training sample is defined based on the prediction error, wherein:

[0027] y conf，label，i =exp(-α·MSE i ),

[0028] Among them, α is a predetermined coefficient, y conf，label，i is the confidence label corresponding to the i-th training sample;

[0029] Use the prediction error loss L MSE , confidence loss L conf and L2 regularization loss L L2To define the total loss function L of the neural network model total ,in:

[0030] L total =L MSE +βL conf +L L2 , where β is a predetermined weight coefficient;

[0031]

[0032] Among them, y conf，i is the confidence output by the neural network model for the i-th training sample;

[0033] Among them, λ is the regularization coefficient, φ j is the jth model parameter of the neural network model, N params is the number of all parameters contained in the neural network model;

[0034] Based on the total loss function L total Calculate the model parameter φ j Gradient in:

[0035]

[0036] is the model parameter φ j Based on the L MSE The gradient of is the model parameter φ j Based on the L conf The gradient of The model parameter φ j Based on the L L2 The gradient of

[0037] Update the model parameters φ using gradient descent or Adam optimizer j :

[0038]

[0039] Where η is the learning rate.

[0040] Preferably, in the method for estimating motion parameters between video stream frames, the number of selected valid feature point pairs is n, and the second affine transformation matrix M is obtained using the selected valid feature point pairs. correction The steps include:

[0041] Step a, randomly selecting m point pairs from the n valid feature point pairs, where m is greater than or equal to 3;

[0042] Step b, using the m point pairs to estimate the fourth affine transformation matrix M k ;

[0043] Step c, using the fourth affine transformation matrix M k The reprojection error is calculated for each of the n valid feature point pairs, where:

[0044]

[0045] where e i，k is the reprojection error calculated for the ith valid feature point pair among the n valid feature point pairs, x i and i is the x-axis coordinate and y-axis coordinate of the i-th valid feature point pair in the previous frame adjacent to the current frame, x′ i and y′ i are the coordinates of the i-th valid feature point pair in the x-axis direction and the y-axis direction on the intermediate frame, and ||·|| represents the norm;

[0046] Step d: Calculate the median error of the reprojection error of all n valid feature point pairs k ,in:

[0047]

[0048] Step e, repeating steps a to d for a predetermined K times, and selecting a fourth affine transformation matrix M corresponding to the minimum median error from the obtained K median errors. k As a candidate affine transformation matrix M optical ;

[0049] Step f, using the candidate affine transformation matrix M optical Calculate the reprojection error corresponding to each valid feature point pair among all n valid feature point pairs, where:

[0050]

[0051] Among them, e i is the candidate affine transformation matrix M optical Calculate the reprojection error of the i-th valid feature point pair, i = 1...n;

[0052] Step g, determine whether there is an outlier, where e iThe valid feature point pairs whose values ​​are greater than or equal to the predetermined threshold are determined as outliers; if not, the candidate affine transformation matrix M is output optical is the second affine transformation matrix M correction If yes, then remove the outliers, and use a predetermined estimation algorithm to determine the inliers in the remaining valid feature point pairs, and use the inliers to estimate the second affine transformation matrix M correction .

[0053] Preferably, in the method for estimating motion parameters between video stream frames, the step of using point pairs or interior points to estimate the corresponding affine transformation matrix M comprises: inputting an original point set and a target point set, wherein the point set corresponding to the previous frame adjacent to the current frame in the point pairs or the interior points is determined as the original point set and the point set corresponding to the intermediate frame is determined as the target point set;

[0054] Construct a linear equation based on the original point set and the target point set:

[0055] x′=a*x+b*y+c, y′=d*x+e*y+f,

[0056] Where (x, y) is any source point in the original point set, (x′, y′) is any target point in the target point set, a, b, c, d, e, f are the affine transformation parameters to be solved, where:

[0057]

[0058] Wherein, a is the scaling factor in the x-axis direction, b is the vertical shearing parameter, c is the translation parameter in the x-axis direction, d is the horizontal shearing parameter, e is the scaling factor in the y-axis direction, and f is the translation parameter in the y-axis direction;

[0059] The linear equation is converted into a matrix form, the least square method is used to solve the affine transformation parameters, and an affine transformation matrix M constructed by the affine transformation parameters is obtained.

[0060] Preferably, in the method for estimating motion parameters between video stream frames, each of the first affine transformation matrix, the second affine transformation matrix and the third affine transformation matrix is ​​a matrix M with two rows and three columns,

[0061]

[0062] Wherein, a, b, c, d, e, f are affine transformation parameters contained in the affine transformation matrix, wherein a is the scaling in the x-axis direction, b is the vertical shearing parameter, c is the translation in the x-axis direction, d is the horizontal shearing parameter, e is the scaling in the y-direction, and f is the translation in the y-axis direction;

[0063]

[0064] Among them, a accurate , b accurate , c accurate , d accurate , e accurate , f accurate is the third affine transformation matrix M accurate The corresponding affine transformation parameters;

[0065] Among them, from the third affine transformation matrix M accurate The actual inter-frame motion parameters of the current frame relative to the previous frame adjacent to the current frame extracted from the method include:

[0066] Δx accurate =c accurate , Δx accurate is the actual translation motion parameter in the x-axis direction;

[0067] Δy accurate =f accurate , Δy accurate is the actual translation motion parameter in the y-axis direction;

[0068] Δθ accurate =arctan2(d accurate , a accurate ), Δθ accurate is the actual rotation angle; Δs accurate is the actual scaling factor;

[0069] sh accurate is the actual horizontal shearing parameter;

[0070] sv accurate is the actual vertical shearing parameter.

[0071] On the other hand, a video stabilization method implemented by using any of the above-mentioned methods for estimating motion parameters between video stream frames is provided.

[0072] Preferably, the video stabilization method further comprises: using a predetermined smoothing algorithm and actual inter-frame motion parameters corresponding to a second predetermined number of frames to smooth the motion trajectory of the second predetermined number of frames to obtain smoothed inter-frame motion parameters after smoothing; and using the smoothed inter-frame motion parameters to generate a smoothed affine transformation matrix M smoothed ; The smooth affine transformation matrix M smoothedActing on each frame of the second predetermined number of frames, obtaining a stable video frame corresponding to the second predetermined number of frames; wherein, when the smooth affine transformation matrix M smoothed Acting on the current frame I current When the stable current frame I is obtained current_result ,in:

[0073] I current_result =M accurate *I current *M smoothed .

[0074] On the other hand, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method for estimating motion parameters between frames of a video stream as described above are implemented.

[0075] The above technical solution has the following technical effects:

[0076] The embodiment of the present invention uses a neural network model to make a rough prediction of the inter-frame motion parameters in one stage and uses local optical flow to make accurate estimation of the inter-frame motion parameters, thereby realizing two-stage estimation of the inter-frame motion parameters, improving the estimation efficiency while ensuring the estimation accuracy; moreover, the embodiment of the present invention can adapt to different scenes and motion modes through online adaptive learning of the inter-frame motion parameters, namely the jitter parameters, thereby realizing continuous optimization of the inter-frame motion parameters, and adapting to different video contents and jitter modes, thereby improving the stability and reliability of the estimation algorithm. The embodiment of the present invention can more effectively handle complex vibrations in industrial environments, significantly improve the video anti-shake effect and efficiency, and significantly improve the image quality and stability of cameras on large-scale equipment in ports, which can not only improve the visual effect, but also improve the safety and efficiency of operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A schematic diagram of a flow chart of a method for estimating motion parameters between video stream frames according to an embodiment of the present invention;

[0078] Figure 2 A schematic diagram of obtaining the affine transformation of the intermediate frame in an estimation method according to an embodiment of the present invention;

[0079] Figure 3 In the estimation method according to an embodiment of the present invention, the third affine transformation matrix M accurate Schematic diagram of the estimation;

[0080] Figure 4 Schematic diagram of a brief flow of a method for estimating motion parameters between video stream frames according to the embodiment of the present invention;

[0081] Figure 5 for Figure 4Detailed expansion process diagram;

[0082] Figure 6 A schematic diagram of the working process of a neural network model in an estimation method according to an embodiment of the present invention;

[0083] Figure 7 In an estimation method according to an embodiment of the present invention, the intermediate frame I is obtained by using the feature point pair warp Relative to the previous frame I previous The second affine transformation matrix M correction Schematic diagram of the principle;

[0084] Figure 8 A schematic diagram of a process for estimating an affine transformation matrix between two frames of images using feature point sets of two frames, such as valid feature point pairs, in an estimation method according to an embodiment of the present invention;

[0085] Fig. 9 The figure is a flow chart of realizing online updating of neural network model parameters in one embodiment of the present invention. DETAILED DESCRIPTION

[0086] To further illustrate the various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, a person of ordinary skill in the art should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0087] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.

[0088] Embodiment 1:

[0089] Figure 1 FIG. 1 is a flow chart of a method for estimating motion parameters between video stream frames according to an embodiment of the present invention. Figure 1 The method for estimating motion parameters between video stream frames of this embodiment includes the following steps:

[0090] Acquire the current frame in real time; in a specific implementation, the current frame is the current frame image in the video currently being processed; wherein the video is composed of multiple image frames with time information; extract the color image frame of the latest acquisition time from the video as the initial image frame, grayscale the initial image frame, obtain the grayscale current frame, that is, obtain the grayscale image of the current frame; in subsequent processing, use the grayscale image of the current frame for processing;

[0091] According to the actual inter-frame motion parameters between each frame in the N frames of images before the current frame and the previous frame adjacent to itself, the translation trajectory of the N frames of images in the predetermined x-axis direction, the translation trajectory of the N frames of images on the predetermined y-axis, the rotation trajectory of the N frames of images and the scaling trajectory of the N frames of images, a pre-trained neural network model is used to predict the predicted inter-frame motion parameters of the current frame relative to the previous frame adjacent to the current frame, and the inter-frame motion parameters include: a translation motion parameter Δx in the predetermined x-axis direction, a predetermined translation parameter Δy in the predetermined y-axis direction, a rotation angle Δθ and a scaling factor Δs; preferably, the neural network model is a neural network model with online learning capability;

[0092] Generate a first affine transformation matrix of a current frame relative to a previous frame adjacent to the current frame based on the predicted inter-frame motion parameter;

[0093] Apply the first affine transformation matrix to the current frame to obtain a corrected intermediate frame, where I warp =M predict *I current , I current is the current frame, I warp is the corrected intermediate frame, M predict is the first affine transformation matrix; Figure 2 A schematic diagram of affine transformation to obtain the above intermediate frame;

[0094] In a specific implementation, the above steps may be referred to as a jitter prediction stage, i.e., a stage of rough jitter correction based on a deep learning neural network; specifically, the steps may be implemented by a program module capable of executing the corresponding steps, such as a jitter prediction module;

[0095] Using a predetermined feature point detection algorithm to perform feature point detection on a grayscale image of a previous frame adjacent to the current frame, a plurality of target feature points are obtained, and using a selected optical flow method to calculate the relative motion of the plurality of target feature points between the current frame and the previous frame adjacent to the current frame; in a specific implementation, the predetermined feature point detection algorithm includes: Harris corner point or Shi-Tomasi corner point detection algorithm; in a specific implementation, the selected optical flow method is a pyramid optical flow method (PyramidalLucas-Kanade);

[0096] According to the relative motion of the plurality of target feature points between the current frame and the frame before the current frame, a feature point pair having consistent motion and satisfying a predetermined condition is selected as a valid feature point pair, wherein the predetermined condition includes: a predetermined motion consistency condition and / or a predetermined stability condition;

[0097] Use the selected valid feature point pairs to obtain the second affine transformation matrix M of the intermediate frame relative to the previous frame adjacent to the current frame correction , and according to the second affine transformation matrix Mcorrection Determine the third affine transformation matrix M of the current frame relative to the previous frame adjacent to the current frame accurate ,in:

[0098] I previous =M correction *I warp =M correction *M predict *I current ; Among them, I previous is the previous frame adjacent to the current frame;

[0099] M accurate =M correction *M predict ;

[0100] I previous =M accurate *I current ;

[0101] Figure 3 is the third affine transformation matrix M accurate That is, the estimation diagram of the exact affine transformation matrix; from the third affine transformation matrix M accurate The actual inter-frame motion parameters of the current frame relative to the previous frame are extracted; the steps from the feature point detection to the actual inter-frame motion parameters are a two-stage precise jitter correction stage based on feature point detection and optical flow method, which can be specifically implemented by a program module that can execute the corresponding steps, such as a precise jitter correction module; the difference between the actual inter-frame motion parameters and the predicted inter-frame motion parameters is obtained, the difference between the actual inter-frame motion parameters and the predicted inter-frame motion parameters is stored, and when the number of stored differences reaches a first predetermined number, the parameters of the neural network model are updated online. Preferably, the first predetermined number = N. The above-mentioned online update of the parameters of the neural network model can be specifically implemented by a program module that can execute the corresponding steps, such as an online learning update module.

[0102] In a specific implementation, the horizontal and vertical shear parameters are ignored in the affine transformation, and the corresponding affine transformation matrix M can be constructed by the translation parameters in the known inter-frame motion parameters, namely, the translation amounts Δx and Δy, the rotation angle Δθ and the scaling factor Δs; wherein the corresponding affine transformation matrix generated based on the predicted inter-frame motion parameters is:

[0103]

[0104] Embodiment 2:

[0105] Figure 4 The following is a schematic flow chart of a method for estimating motion parameters between video stream frames according to the embodiment of the present invention, wherein each step in the flow chart is implemented by a corresponding program module. Figure 5for Figure 4 Detailed expansion process diagram. Figure 4 and Figure 5 In addition to the jitter prediction module, the precise jitter correction module, and the online learning update module, this embodiment also includes the steps of system initialization, smoothing the inter-frame motion parameters, i.e., jitter parameters, and resource release. Specifically:

[0106] Before starting estimation using the estimation method of an embodiment of the present invention, a system initialization step is also included, specifically including: initializing the camera device to ensure that it can capture the video stream normally; such as including: starting the camera, capturing the initial frame, etc.; initializing the neural network model, pre-configuring a pre-trained and online learning neural network model, aiming to predict the motion parameters between frames, including translation motion parameters (Δx, Δy), rotation angle (Δθ) and scaling factor (Δs); initializing the feature point detector, selecting an appropriate feature point detection algorithm such as Harris corner point or Shi-Tomasi corner point detection method, to facilitate subsequent feature point tracking.

[0107] Preferably, in this embodiment, after obtaining the actual inter-frame motion parameters corresponding to the corresponding frames, it also includes caching the above-mentioned actual inter-frame motion parameters, and when the number of video frames corresponding to the cached actual inter-frame motion parameters reaches a predetermined number, it also includes: using a predetermined smoothing algorithm and the actual inter-frame motion parameters corresponding to the predetermined number of frames, smoothing the motion trajectory of the predetermined number of frames to obtain the smoothed inter-frame motion parameters after smoothing; in a specific implementation, the predetermined smoothing algorithm includes: a simple moving average method; in a specific implementation, the above-mentioned motion trajectory includes: a translation trajectory, a rotation trajectory and / or a scaling trajectory; using the smoothed inter-frame motion parameters to generate a smoothed affine transformation matrix M smoothed ; Smooth affine transformation matrix M smoothed Acting on each of the predetermined number of frames, obtaining a stable video frame corresponding to the predetermined number of frames; wherein, when the smooth affine transformation matrix M smoothed Acts on the current frame I current When the stable current frame I is obtained current_result ,in:

[0108] I current_result =M accurate *I current *M smoothed .

[0109] Preferably, after the stabilized video is output, the anti-shake or video stabilization function is terminated; then, resources are released, video capture is terminated, the camera is turned off, and all occupied system resources are released. Among them, the steps of jitter prediction, estimation and correction, online update of neural network model parameters, and smoothing of jitter parameters are performed in a loop; in specific implementation, this can be achieved through a loop structure.

[0110] In a specific implementation, the affine transformation mentioned in the embodiment of the present invention includes scaling, rotation and translation operations; each of the affine transformation matrices mentioned, such as the first affine transformation matrix, the second affine transformation matrix and the third affine transformation matrix, is a matrix M with two rows and three columns,

[0111]

[0112] Wherein, a, b, c, d, e, f are affine transformation parameters contained in the affine transformation matrix, wherein a is the scaling in the predetermined x-axis direction, b is the vertical shearing parameter, c is the translation in the predetermined x-axis direction, d is the horizontal shearing parameter, e is the scaling in the predetermined y-axis direction, and f is the translation in the predetermined y-axis direction;

[0113]

[0114] Among them, a accurate , b accurate , c accurate , d accurate , e accurate , f accurate is the third affine transformation matrix M accurate The corresponding affine transformation parameters;

[0115] Among them, from the third affine transformation matrix M accurate The actual inter-frame motion parameters of the current frame relative to the previous frame adjacent to the current frame extracted from include:

[0116] Δx accurate =c accurate , Δx accurate is the actual translation motion parameter in the predetermined x-axis direction;

[0117] Δy accurate =f accurate , Δy accurate is the actual translation motion parameter in the predetermined y-axis direction;

[0118] Δθ accurate =arctan2(d accurate , a accurate ), Δθ accurate is the actual rotation angle;

[0119] Δs accurate is the actual scaling factor;

[0120] sh accurate is the actual horizontal shearing parameter;

[0121] sv accurate is the actual vertical shearing parameter.

[0122] The following describes in detail the specific implementations of the neural network model used in the embodiments of the present invention to predict inter-frame motion parameters, use local optical flow to accurately estimate inter-frame motion parameters, and calculate the actual affine transformation matrix.

[0123] The input of the neural network model is time series data, with a total of 8 dimensions, including:

[0124] For the current frame, the actual inter-frame motion parameters between each frame in the past N frames and the previous frame adjacent to it include: translation parameters Δx and (Δy), rotation angle (Δθ) and scaling factor (Δs); the trajectory of the accumulated amount includes: the translation trajectory of the past N frames in the x-axis direction, the translation trajectory in the y-axis direction, the rotation trajectory and the scaling trajectory, wherein the translation trajectory, the rotation trajectory and the scaling trajectory are determined by the following formula:

[0125] x(t)=x(t-1)+Δx(t), x(0)=0,

[0126] y(t)=y(t-1)+Δy(t), y(0)=0,

[0127] θ(t-1)+Δθ(t), θ(0)=0,

[0128] s(t)=s(t-1)×Δs(t), s(0)=1,

[0129] Among them, x(t) is the position in the x-axis direction at time t, such as the horizontal position, and is the accumulated horizontal position, which is used to describe the translation trajectory of the camera in the x-direction; y(t) is the position in the y-axis direction at time t, such as the vertical position, and is the accumulated vertical position, which is used to describe the translation trajectory of the camera in the y-direction; θ(t) is the rotation angle at time t, and is the accumulated rotation angle, which is used to describe the rotational motion of the camera; s(t) is the zoom factor at time t, and is the accumulated zoom factor, which is used to describe the zoom motion of the camera; Δx(t) is the translation in the x-axis direction at time t, such as the displacement in the horizontal direction; Δy(t) is the translation in the y-direction at time t, such as the displacement in the vertical direction; Δθ(t) is the change in the rotation angle at time t, and Δs(t) is the change in the zoom factor at time t;

[0130] The outputs of the neural network model include:

[0131] The predicted inter-frame motion parameters of the current frame relative to the previous frame adjacent to the current frame, that is, the first affine transformation matrix M predict The corresponding predicted inter-frame motion parameters; and the confidence probability of this prediction.

[0132] In this embodiment, the neural network model includes an input layer, a first LSTM layer, a second LSTM layer, a dense layer and an output layer connected in sequence, and the output layer includes a mean output layer and a confidence output layer; wherein the input layer is used to receive input parameters of a predetermined time step; the first LSTM layer is used to process the data output by the input layer and return the output of all time steps; the second LSTM layer is used to process the data output by the first LSTM layer and only return the output of the last time step; the dense layer is used to process the data output by the second LSTM layer using a nonlinear transformation; the mean output layer is used to process the data output by the dense layer and output the mean of each motion parameter in the predicted inter-frame motion parameter; the confidence output layer is used to process the data output by the dense layer and output the confidence probability corresponding to this prediction.

[0133] Figure 6 FIG. 1 is a schematic diagram of the workflow of a neural network model in an estimation method according to an embodiment of the present invention. Figure 6 In this embodiment, the input layer receives 8-dimensional data of 150 time steps; the data first passes through an LSTM layer that returns a sequence, which contains 64 units in this example; then passes through another LSTM layer, which contains 32 units and does not return a sequence; followed by a dense layer with a ReLU activation function, which contains 32 neurons in this example; and finally divided into two output branches: one for predicting the mean of 4 inter-frame motion parameters (Δx, Δy, Δθ, Δs); the other for predicting the overall confidence, using the Sigmoid activation function in a specific implementation. This structure allows the model to learn the characteristics of time series data and finally output the predicted parameter mean and an overall confidence assessment. The dense layer provides additional nonlinear transformation capabilities after the LSTM layer, which helps the model capture more complex patterns.

[0134] More specifically, each layer of the neural network model is:

[0135] InputLayer

[0136] Name: input_layer_2; Output shape: (None, 150, 8); Number of parameters: 0; Description: This layer receives input data; None means the batch size is variable, 150 is the number of time steps, and 8 is the number of features per time step; this means that each input sequence contains 150 time points, and each time point has 8 features.

[0137] The first LSTM layer

[0138] Name: lstm7; Output shape: (None, 150, 64); Number of parameters: 18688; Connected to: input_layer_2; Description: This is an LSTM layer that returns the complete sequence; It processes the input sequence and outputs the state of 64 units for each time step; The number of parameters is calculated due to the internal structure of the LSTM, which includes the input gate, forget gate, output gate, and cell state.

[0139] Second LSTM layer

[0140] Name: lstm_8; Output shape: (None, 32); Number of parameters: 12416; Connected to: lstm_7; Description: This LSTM layer returns only the output of the last time step; it compresses the output sequence of the first LSTM layer into a 32-dimensional vector; this layer has fewer parameters because the input dimension is reduced, such as from 64 to 32).

[0141] Dense

[0142] Name: dense_3; Output shape: (None, 32); Number of parameters: 1056; Connected to: lstm_8; Description: This is a fully connected layer with 32 neurons; It receives the output of the second LSTM layer and applies a nonlinear transformation, which can be a ReLU activation function in specific implementations; The number of parameters is 32*32+32, that is, the weight matrix plus the bias term.

[0143] Mean output layer (Dense)

[0144] Name: means; Output shape: (None, 4); Number of parameters: 132; Connects to: dense_3[0]; Description: This layer outputs 4 predictions, corresponding to Δx, Δy, Δθ, Δs; The number of parameters is 32*4+4, i.e. the weight matrix plus the bias term.

[0145] Confidence output layer (Dense)

[0146] Name: confidence; Output shape: (None, 1); Number of parameters: 33; Connected to: dense_3; Description: This layer outputs a single confidence value; In a specific implementation, it uses the Sigmoid activation function to ensure that the output is between 0 and 1; Among them, the number of parameters is 32*1+1, that is, the weight matrix plus the bias term.

[0147] The structure of the neural network model allows the model to learn the characteristics of time series data and finally output the predicted parameter mean and confidence assessment; the dense layer provides additional nonlinear transformation capabilities after the LSTM layer, which helps the model capture more complex patterns; the total number of parameters is 32325, and all parameters can be optimized through training.

[0148] In general, this network first uses two LSTM layers to process time series data and capture time dependencies. It then uses a dense layer for further feature extraction and is finally divided into two output branches: one for predicting the mean of the four parameters and the other for predicting the overall confidence. This structure allows the model to capture the complex patterns of time series while providing uncertainty estimates for the predictions. In a specific implementation, the total number of parameters: 32325; the number of trainable parameters: 32325; the number of non-trainable parameters: 0. This indicates that all parameters can be optimized through training, and there are no frozen or fixed parameters.

[0149] In a specific implementation, in the precise shake correction stage, the detailed process of sparse optical flow calculation based on the pyramid selected optical flow method and accelerated by OpenCL is as follows:

[0150] Input Validation:

[0151] Check whether the minimum eigenvalue needs to be calculated, if yes, return false; check whether the device supports OpenCL acceleration, if not, return false; check whether the image sizes of the previous and next frames are consistent, if not, return false; check whether the image type is a single-channel 8-bit image, if not, return false; check the previous frame I previous Check whether the feature point is empty, whether the type is float32 and whether it is stored continuously; if not, return false; check whether the size of the feature point is 1 row or 1 column, if not, return false.

[0152] Initial flow verification:

[0153] If the initial stream is used, check the current frame I current Whether the feature points are empty, whether the type is float32 and whether they are stored continuously; if not, return false; check the current frame I currentIs the number of feature points in the same frame as the previous frame I previous If the number of feature points in the frame is not consistent, false is returned. If the initial stream is not used, the previous frame I is created. previous The feature points in the current frame I have the same size and type. current Feature points.

[0154] Parameter check:

[0155] Call the checkParam function to check the validity of other parameters and return false if they are invalid.

[0156] Create the error matrix:

[0157] If the error needs to be output, create an error matrix umatErr of the size of the number of feature points, the type is float32. If the error does not need to be output, also create an error matrix umatErr of the size of the number of feature points.

[0158] Create the state matrix:

[0159] Create a status matrix umatStatus of size equal to the number of feature points, of type uint8.

[0160] Copy feature points:

[0161] Copy the previous frame I_previous feature points to umatPrevPts.

[0162] Sparse optical flow calculation:

[0163] Call the sparse function to calculate the sparse optical flow using the previous and next frame images, feature points, state matrix, and error matrix.

[0164] Returns:

[0165] Returns the Boolean value of the calculation result.

[0166] Figure 7 The intermediate frame I is obtained by using feature point pairs. warp Relative to the previous frame I previous The second affine transformation matrix M correction Schematic diagram of the principle. Figure 7 , where point x and point d are I previous The feature points in the graph are point x' and point d'. current The corresponding feature points in ; points x and x', points d and d' are feature point pairs, such as valid feature point pairs. By tracking the above two two-dimensional point sets, the affine matrix M can be calculated correction , the affine matrix M correctionThe first point set {x, d} can be aligned with the second point set {x', d'} as much as possible after the affine transformation.

[0167] Figure 8 FIG. 1 is a flow chart of estimating the affine transformation matrix between two frames of images using feature point sets of two frames, such as valid feature point pairs, in an estimation method according to an embodiment of the present invention. Figure 8 , the general process of obtaining the affine transformation matrix based on the feature point set of the previous and next two frames of images includes:

[0168] Input the original point set and the target point set, which can be provided by the user; the original point set is the feature point set in the previous frame image, i.e., the source point set, and the target point set is the feature point set in the next frame image;

[0169] Construct a linear equation system: Based on the input point set, construct the following linear equation:

[0170] x′=a*x+b*y+c,y′=d*x+e*y+f,where (x, y) is the source point, (x′, y′) is the target point, and a, b, c, d, e, f are the affine transformation parameters in the affine transformation matrix to be solved;

[0171]

[0172] Wherein, M is the affine transformation matrix to be solved; a is the scaling factor in the x-axis direction, b is the vertical shear parameter, c is the translation parameter in the x-axis direction, d is the horizontal shear parameter, e is the scaling factor in the y-axis direction, and f is the translation parameter in the y-axis direction;

[0173] The linear equation is converted into a matrix form A*X=B to solve the vector X consisting of the affine transformation parameters using linear algebra methods. Specifically, the above matrix form corresponds to:

[0174]

[0175] Here, A is a matrix constructed by the two-dimensional coordinates of the source points in the original point set, X is the affine transformation parameter vector to be solved, and B is the coordinates of the target point; the original point set and the target point set each contain n feature points; the distance from point (x1, y1) to point (x n ,y n ) is a point in the original point set; point (x1′, y1′) to point (x n ′,y n ′) is the distance from point (x1, y1) in the target point set to point (x n ,y n ) corresponding to the point.

[0176] The matrix X is solved using the least squares method, where the matrix X is solved by the least squares method to minimize the sum of squares of the residuals and obtain X=(A T *A) (-1) *A T *B;

[0177] Construct an affine transformation matrix M: Construct a 2x3 affine transformation matrix M based on the solved matrix X;

[0178] Because there may be dynamic objects or matching errors in the actual image, which will cause some feature points to not conform to the overall transformation relationship; therefore, it is necessary to identify and remove these outliers. In a specific implementation, outliers are determined based on reprojection errors. If it is determined that there are outliers, a selected robust estimation method, such as the Random Sample Consensus (RANSAC), the Least Median of Squares (LMEDS) or the RHO algorithm, is used to determine the inliers in the remaining point set after the outliers are removed, and then all the inliers are used to re-estimate the affine transformation matrix to obtain a more accurate result; in a specific implementation, this step can be re-estimated using the least squares method; if there are no outliers, the calculated affine transformation matrix M is directly output.

[0179] The specific steps of obtaining the affine transformation matrix include:

[0180] Step a, random sampling, randomly selecting m point pairs from n point pairs, for example, for affine transformation, m is greater than or equal to 3; the minimum number of point pairs required for affine transformation is m=3; n is the total number of corresponding point pairs, and its specific value is determined by the input data;

[0181] Step b: Use these m point pairs to estimate an affine transformation matrix M k In the specific implementation, the least square method described above is used to solve; k is a letter used to mark the affine transformation matrix M k ;

[0182] Step c, use the affine transformation matrix M k For all n point pairs, calculate the reprojection error e of the i-th point pair as a valid feature point pair i，k :

[0183]

[0184] Among them, x i and i is the source point coordinates corresponding to the i-th feature point pair, such as the horizontal and vertical coordinates in the previous frame of the current frame, that is, the coordinates in the x-axis direction and the coordinates in the y-axis direction, x′ iand y′ i is the coordinates of the target point corresponding to the i-th feature point pair, such as the horizontal and vertical coordinates on the intermediate frame, and ||·|| represents the norm;

[0185] Step d, calculate the median error of the reprojection error of all n point pairs k :

[0186]

[0187] Step e, repeating steps ad for a total of K times, where K is a predetermined number of times, and selecting the affine transformation matrix corresponding to the minimum median error from the obtained K median errors as the candidate affine transformation matrix M optical That is, the optimal affine matrix estimated by the LMEDS algorithm has both accuracy and robustness, where:

[0188]

[0189] The meaning of this formula is: find the median error median k The smallest k*th corresponding affine transformation matrix M k *As M optical ; Where K is the pre-selected number of iterations, usually a large value such as 100 or 1000 is selected to ensure that there is a high enough probability of finding a good affine transformation matrix. In a specific implementation, K = 100;

[0190] Step f, use the candidate affine transformation matrix M optical Calculate the reprojection error corresponding to each valid feature point pair among all n valid feature point pairs, where:

[0191]

[0192] Among them, e i is the candidate affine transformation matrix M optical Calculate the reprojection error of the i-th valid feature point pair, i = 1...n;

[0193] Step e, determine whether there is an outlier, where e i A valid feature point pair whose value is greater than or equal to a predetermined threshold is determined as an outlier; preferably, the predetermined threshold is 2.5σ, where σ is the standard deviation of the error.

[0194] The median is a statistical concept, which refers to the value in the middle after a set of data is arranged in order of size. n}, the median is defined as:

[0195]

[0196] In this example, the least median square (LMEDS) method can effectively estimate the affine transformation matrix while being robust to outliers. LMEDS is particularly suitable for situations where there are many outliers because the median is not easily affected by extreme values.

[0197] In one embodiment of the present invention, extracting each inter-frame motion parameter from a given affine transformation matrix M comprises the following steps:

[0198]

[0199] Extract the translation parameter Δx in the x-axis direction and the translation parameter Δy in the y-axis direction, where:

[0200] Δx=c,Δy=f,

[0201] The rotation angle Δθ is calculated by the following formula:

[0202] Δθ=arctan2(d,a),

[0203] The arctan2 function is used here to ensure that the correct quadrant is obtained;

[0204] The scaling parameter Δs is obtained by calculating the square root of the determinant of the 2x2 submatrix in the upper left corner of the matrix, wherein in the application scenario of the present invention, no negative number will appear in the square root sign;

[0205]

[0206] The horizontal shear parameter (sh) is calculated by the following formula:

[0207]

[0208] The vertical shear parameter sv is calculated by the following formula:

[0209]

[0210] It should be noted that this decomposition assumes that the affine transformation can be decomposed into a combination of scaling, rotation, shearing and translation; in some cases, especially when the matrix is ​​close to singularity, this decomposition may be unstable or non-unique; in the scenario of the embodiment of the present invention, the jitter matrix will not appear singular, so there is no need to consider this situation.

[0211] In one embodiment of the present invention, when training a neural network model, labels are created for confidence outputs.

[0212] Confidence output has multiple values ​​in the video stabilization system: it quantifies the uncertainty of the prediction, supports intelligent decision-making, improves the system's adaptability and anomaly detection capabilities, and promotes the model's self-evaluation and optimization; the embodiment of the present invention trains the model to output confidence through a carefully designed loss function, rather than directly providing labels, so that the model can naturally learn, adapt to complex environments, and improve generalization capabilities. This method significantly enhances the performance, robustness, and adaptability of the system, enabling it to handle various complex situations more intelligently, thereby providing a more reliable video stabilization effect. Without confidence output, the system may face problems such as blindly trusting predictions, unstable performance, and difficulty adapting to complex environments, affecting the overall effect and reliability.

[0213] In the specific implementation, since confidence is a quantification of the uncertainty of the model in its prediction, an indirect method is used to define these labels instead of directly providing the true value of the confidence.

[0214] In a specific implementation, the steps of training the neural network model include:

[0215] For each training sample in the N training samples, that is, each prediction sample, the predicted inter-frame motion parameter (Δx pred,i , Δy pred,i , Δθ pred,i , Δs pred,i ) and the actual inter-frame motion parameters of the training samples (Δx true，i , Δy tru,i , Δθ true，i , Δs true，i ), and use the mean square error as the prediction error, where:

[0216] Among them, MSE i is the mean square error of the i-th training sample;

[0217] Define the confidence label y corresponding to the i-th training sample based on the prediction error conf,label，i ,in:

[0218] y conf，label，i =exp(-α·MSE i ),

[0219] Among them, α is a predetermined coefficient; α is used to control the relationship between confidence and error; in specific implementation, its value is predetermined by experiment, for example, α = 1; this definition makes the confidence close to 1 when the prediction error is small; when the error is large, the confidence is close to 0;

[0220] Use the prediction error loss L MSE , confidence loss Lconf and L2 regularization loss L L2 To define the total loss function L of the neural network model total ,in:

[0221] L total =L MSE +βL conf +L L2 , where β is a predetermined weight coefficient, which is used to balance L MSE and L conf In one specific implementation, β is set to 1. In other implementations, the value of β can be adjusted according to actual conditions.

[0222]

[0223] Among them, y conf，i is the confidence output by the neural network module for training sample i;

[0224] Among them, in order to enable the neural network model to output high confidence when the prediction is accurate and low confidence when the prediction is inaccurate, the confidence loss L is introduced conf , to encourage a consistent relationship between confidence and prediction error; in this example, the mean squared error is used to measure the accuracy of the confidence prediction:

[0225] Among them, λ is the regularization coefficient, φ j is the jth model parameter of the neural network model, N params is the number of all parameters contained in the neural network model;

[0226] Based on the total loss function L total Calculate the model parameter φ j Gradient To achieve the use of back propagation algorithm to update the neural network model parameters, where:

[0227]

[0228] is the model parameter φ j Based on L MSE The gradient of is the model parameter φ j Based on L conf The gradient of Model parameter φ j Based on L L2 The gradient of

[0229] Update the model parameters φ using gradient descent or Adam optimizer j :

[0230]

[0231] Where η is the learning rate.

[0232] Fig. 9 FIG. 1 is a flow chart of implementing online updating of neural network model parameters in one embodiment of the present invention. Fig. 9 ,The online update of neural network model parameters specifically includes the following steps:

[0233] 1. Initialization phase:

[0234] Start: The starting point of the process;

[0235] Initialize the sliding window buffer: set the buffer capacity to a predetermined number of frames, such as 100 frames, for storing the latest motion parameter data;

[0236] 2. Data processing stage:

[0237] Receive new data: obtain the inter-frame motion parameters corresponding to the current frame, that is, the motion parameter increment (Δx, Δy, Δθ, Δs); calculate the cumulative trajectory: update the cumulative trajectory (x, y, θ, s) according to the stored data of the previous frame and the increment of the current frame, where x is the accumulated translation trajectory in the x-axis direction; y is the accumulated translation trajectory in the y-axis direction, θ is the rotation trajectory, and s is the scaling trajectory; specifically, the formula used for updating is:

[0238] x=x+Δx; y=y+Δy; θ=θ+Δθ; s=s+Δs.

[0239] 3. Buffer Management:

[0240] If the buffer is full, remove the oldest data; add new data and accumulated trajectories to the buffer; check the update condition: for example, reaching a certain number N of new samples, such as N = 600 batches, to trigger the model update.

[0241] 4. Model update phase:

[0242] Sampling from the buffer: Randomly sample a batch of data for training to avoid the bias caused by sequential data;

[0243] Model forward propagation: input sample data and generate predicted values ​​and confidence levels;

[0244] Define confidence labels: Calculate confidence labels based on prediction errors;

[0245] Calculate the total loss L total : Combine MSE loss, confidence loss and L2 regularization loss to form the total loss;

[0246] Back propagation: According to the total loss L total Calculate the gradient;

[0247] Update model parameters: Apply gradients to update model weights;

[0248] Elastic Weight Convergence (EWC): prevents catastrophic forgetting of the model and retains important weights;

[0249] Learning rate decay: reduce the learning rate every 1000 iterations to improve training stability; gradually reduce the learning rate as training progresses, which helps the model converge more stably in the later stages of training;

[0250] Reset update condition counter: After the model parameters are updated, reset the counter and continue to receive new data.

[0251] Through the above steps, the online learning update module can efficiently and stably optimize the neural network model to adapt to different video content and jitter patterns, thereby significantly improving the performance and reliability of the video stabilization system.

[0252] Embodiment three:

[0253] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method described above. Although the present invention is specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes in form and details may be made to the present invention without departing from the spirit and scope of the present invention as defined in the appended claims, and all of these are within the scope of protection of the present invention.

Claims

1. A method for estimating motion parameters between video stream frames, characterized in that: include: Get the current frame in real time; According to the actual inter-frame motion parameters between each frame in the N frames of images before the current frame and the previous frame adjacent to the current frame, the translation trajectory of the N frames of images in a predetermined x-axis direction, the translation trajectory of the N frames of images in a predetermined y-axis direction, the rotation trajectory of the N frames of images and the scaling trajectory of the N frames of images, a predicted inter-frame motion parameter of the current frame relative to the previous frame adjacent to the current frame is predicted using a pre-trained neural network model, wherein the inter-frame motion parameters include: a translation motion parameter Δx in a predetermined x-axis direction, a translation motion parameter Δy in a predetermined y-axis direction, a rotation angle Δθ and a scaling factor Δs; Generate a first affine transformation matrix of the current frame relative to a previous frame adjacent to the current frame based on the predicted inter-frame motion parameter; Apply the first affine transformation matrix to the current frame to obtain a corrected intermediate frame, where I warp =M predict *I current , I current is the current frame, I warp is the corrected intermediate frame, M predict is the first affine transformation matrix; Using a predetermined feature point detection algorithm to perform feature point detection on a grayscale image of a previous frame adjacent to the current frame to obtain a plurality of target feature points, and using a selected optical flow method to calculate the relative motion of the plurality of target feature points between the current frame and the previous frame adjacent to the current frame; According to the relative motion of the plurality of target feature points between the current frame and a previous frame adjacent to the current frame, a feature point pair satisfying a predetermined condition is selected as a valid feature point pair, wherein the predetermined condition includes: a predetermined motion consistency condition and / or a predetermined stability condition; Use the selected valid feature point pairs to obtain a second affine transformation matrix M of the intermediate frame relative to the previous frame adjacent to the current frame. correction , and according to the second affine transformation matrix M correction Determine a third affine transformation matrix M of the current frame relative to a previous frame adjacent to the current frame accurate ,in: I previous =M correction *I warp =M correction *M predict *I current ; Among them, I previous is a previous frame adjacent to the current frame; M accurate =M correction *M predict ; I previous =M accurate *I current ; From the third affine transformation matrix M accurate extracting an actual inter-frame motion parameter of the current frame relative to a previous frame adjacent to the current frame; Obtain the difference between the actual inter-frame motion parameters and the predicted inter-frame motion parameters, store the difference between the actual inter-frame motion parameters and the predicted inter-frame motion parameters, and when the number of the stored differences reaches a first predetermined number, update the parameters of the neural network model online.

2. The method for estimating motion parameters between video stream frames according to claim 1, characterized in that: In the step of predicting the predicted inter-frame motion parameters of the current frame relative to the previous frame adjacent to the current frame using a pre-trained neural network model based on the pre-obtained actual inter-frame motion parameters between each frame in the N frames of images before the current frame and the previous frame adjacent to the current frame: The input parameters of the neural network model include: Pre-stored actual inter-frame motion parameters between each frame in the N frames of images and its adjacent previous frame; The translation trajectory of the N frames of images in the x-axis direction, the translation trajectory of the N frames of images in the y-axis direction, the rotation trajectory of the N frames of images, and the scaling trajectory of the N frames of images, wherein the translation trajectory, the rotation trajectory, and the scaling trajectory are determined by the following formula: x(t)=x(t-1)+Δx(t),x(0)=0, y(t)=y(t-1)+Δy(t), y(0)=0, θ(t-1)+Δθ(t),θ(0)=0, s(t)=s(t-1)×Δs(t), s(0)=1, Where x(t) is the position in the x-axis direction at time t; y(t) is the position in the y-axis direction at time t; θ(t) is the rotation angle at time t; s(t) is the scaling factor at time t; Δx(t) is the translation in the x-axis direction at time t; Δy(t) is the translation in the y-axis direction at time t; Δθ(t) is the change in the rotation angle at time t, and Δs(t) is the change in the scaling factor at time t. The outputs of the neural network model include: Predicted inter-frame motion parameters of the current frame relative to a previous frame adjacent to the current frame; and, The confidence probability of this prediction.

3. The method for estimating motion parameters between video stream frames according to claim 2, characterized in that: The neural network model includes an input layer, a first LSTM layer, a second LSTM layer, a dense layer and an output layer connected in sequence, and the output layer includes a mean output layer and a confidence output layer; wherein, The input layer is used to receive the input parameters of the predetermined time step; The first LSTM layer is used to process the data output by the input layer and return the output of all time steps; The second LSTM layer is used to process the data output by the first LSTM layer and only return the output of the last time step; A dense layer, used to process the data output by the second LSTM layer using a nonlinear transformation; A mean output layer, used for processing the data output by the dense layer, and outputting the mean of each motion parameter in the predicted inter-frame motion parameter; The confidence output layer is used to process the data output by the dense layer and output the confidence probability corresponding to this prediction.

4. The method for estimating motion parameters between video stream frames according to claim 1, characterized in that: The steps of training the neural network model include: For each of the N training samples, a mean square error between the predicted inter-frame motion parameters predicted by the neural network model and the actual inter-frame motion parameters is calculated, and the mean square error is used as the prediction error, wherein: where Δx pred,i ,Δy pred,i ,Δθ pred,i ,Δs pred,i is the predicted inter-frame motion parameter of the i-th training sample, Δx true,i ,Δy tru,i ,Δθ true,i ,Δs true,i is the actual inter-frame motion parameter of the i-th training sample, MSE i is the mean square error of the i-th training sample; A confidence label corresponding to the training sample is defined based on the prediction error, wherein: y conf,label,i =exp(-α·MSE i ), Among them, α is a predetermined coefficient, y conf,label,i is the confidence label corresponding to the i-th training sample; Use the prediction error loss L MSE , confidence loss L conf and L2 regularization loss L L2 To define the total loss function L of the neural network model total ,in: L total =L MSE +βL conf +L L2 , where β is a predetermined weight coefficient; Among them, y conf,i is the confidence output by the neural network model for the i-th training sample; Among them, λ is the regularization coefficient, φ j is the jth model parameter of the neural network model, N params is the number of all parameters contained in the neural network model; Based on the total loss function L total Calculate the model parameter φ j Gradient in: is the model parameter φ j Based on the L MSE The gradient of is the model parameter φ j Based on the L conf The gradient of The model parameter φ j Based on the L L2 The gradient of Update the model parameters φ using gradient descent or Adam optimizer j : Where η is the learning rate.

5. The method for estimating motion parameters between video stream frames according to claim 1, characterized in that: The number of selected valid feature point pairs is n, and the second affine transformation matrix M is obtained using the selected valid feature point pairs. correction The steps include: Step a, randomly selecting m point pairs from the n valid feature point pairs, where m is greater than or equal to 3; Step b, using the m point pairs to estimate the fourth affine transformation matrix M k ; Step c, using the fourth affine transformation matrix M k The reprojection error is calculated for each of the n valid feature point pairs, where: where e i,k is the reprojection error calculated for the ith valid feature point pair among the n valid feature point pairs, x i and i is the x-axis coordinate and y-axis coordinate of the i-th valid feature point pair in the previous frame adjacent to the current frame, x′ i and y′ i are the coordinates of the i-th valid feature point pair in the x-axis direction and the y-axis direction on the intermediate frame, and ‖·‖ represents the norm; Step d: Calculate the median error of the reprojection error of all n valid feature point pairs k ,in: Step e, repeating steps a to d for a predetermined K times, and selecting a fourth affine transformation matrix M corresponding to the minimum median error from the obtained K median errors. k As a candidate affine transformation matrix M optical ; Step f, using the candidate affine transformation matrix M optical Calculate the reprojection error corresponding to each valid feature point pair among all n valid feature point pairs, where: Among them, e i is the candidate affine transformation matrix M optical Calculate the reprojection error of the i-th valid feature point pair, i=1...n; Step g, determine whether there is an outlier, where e i The valid feature point pairs whose values ​​are greater than or equal to the predetermined threshold are determined as outliers; if not, the candidate affine transformation matrix M is output optical is the second affine transformation matrix M correction If yes, then remove the outliers, and use a predetermined estimation algorithm to determine the inliers in the remaining valid feature point pairs, and use the inliers to estimate the second affine transformation matrix M correction .

6. The method for estimating motion parameters between video stream frames according to claim 5, characterized in that: The steps of using point pairs or inliers to estimate the corresponding affine transformation matrix M include: Inputting an original point set and a target point set, wherein a point set corresponding to a previous frame adjacent to the current frame among the point pairs or the inner points is determined as the original point set and a point set corresponding to the intermediate frame is determined as the target point set; Construct a linear equation based on the original point set and the target point set: x′=a*x+b*y+c, y′=d*x+e*y+f, Where (x, y) is any source point in the original point set, (x′, y′) is any target point in the target point set, a, b, c, d, e, f are the solved affine transformation parameters, where: Wherein, a is the scaling factor in the x-axis direction, b is the vertical shearing parameter, c is the translation parameter in the x-axis direction, d is the horizontal shearing parameter, e is the scaling factor in the y-axis direction, and f is the translation parameter in the y-axis direction; The linear equation is converted into a matrix form, the least square method is used to solve the affine transformation parameters, and an affine transformation matrix M constructed by the affine transformation parameters is obtained.

7. The method for estimating motion parameters between video stream frames according to claim 1, characterized in that: Each of the first affine transformation matrix, the second affine transformation matrix and the third affine transformation matrix is ​​a matrix M with two rows and three columns, Wherein, a, b, c, d, e, f are affine transformation parameters contained in the affine transformation matrix, wherein a is the scaling in the x-axis direction, b is the vertical shearing parameter, c is the translation in the x-axis direction, d is the horizontal shearing parameter, e is the scaling in the y-direction, and f is the translation in the y-axis direction; Among them, a accurate , b accurate , c accurate , d accurate , e accurate , f accurate is the third affine transformation matrix M accurate The corresponding affine transformation parameters; Among them, from the third affine transformation matrix M accurate The actual inter-frame motion parameters of the current frame relative to the previous frame adjacent to the current frame extracted from the method include: Δx accurate =c accurate ,Δx accurate is the actual translation motion parameter in the x-axis direction; Δy accurate =f accurate ,Δy accurate is the actual translation motion parameter in the y-axis direction; Δθ accurate =arctan2(d accurate ,a accurate ),Δθ accurate is the actual rotation angle; Δs accurate is the actual scaling factor; sh accurate is the actual horizontal shearing parameter; sv accurate is the actual vertical shearing parameter.

8. A video stabilization method implemented by using the method for estimating motion parameters between video stream frames according to any one of claims 1 to 7.

9. The video stabilization method according to claim 8, characterized in that: Also includes: Using a predetermined smoothing algorithm and actual inter-frame motion parameters corresponding to a second predetermined number of frames, smoothing the motion trajectories of the second predetermined number of frames to obtain smoothed inter-frame motion parameters after smoothing; Generate a smooth affine transformation matrix M using the smooth inter-frame motion parameters smoothed ; The smooth affine transformation matrix M smoothed Acting on each frame of the second predetermined number of frames, obtaining a stable video frame corresponding to the second predetermined number of frames; wherein, when the smooth affine transformation matrix M smoothed Acting on the current frame I current When the stable current frame I is obtained current_result ,in: I current_result =M accurate *I current *M smoothed 。 10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for estimating inter-frame motion parameters of a video stream as described in any one of claims 1 to 7 are implemented.

Images