Object motion estimation and tracking method and device based on adaptive kalman filter

Abstract

The application discloses an object motion estimation and tracking method and device based on adaptive Kalman filtering, first, kinematics analysis is carried out based on the wind resistance, rotation and Magnus effect of a flying ball, and a motion equation of the flying ball is accurately established; then, an adaptive Kalman filtering algorithm is designed to solve the motion equation, that is, convolution features are extracted through an FCOS algorithm, a flying ball dynamic tracking is realized based on a KCF tracker, a target three-dimensional pose is reconstructed through a camera projection model and input into an adaptive Kalman filter, the flying ball motion equation is solved, and the robustness to measurement noise and modeling error is improved; finally, aiming at the problem of autonomous field of view loss, a PWC-Net dense optical flow method is designed to realize autonomous field of view dynamic maintenance, support fast and accurate self-perception motion estimation and tracking of the flying ball, and effectively overcome the loss rate and hysteresis of the flying ball tracking.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of moving object perception tracking, and particularly relates to an object motion estimation and tracking method and device based on adaptive Kalman filtering. BACKGROUND

[0002] Motion target based on computer vision research is essential for various applications including robots, intelligent driving and dangerous operation tasks. In this context, the perception tracking of moving targets has attracted extensive attention. Its focus is to realize the target detection, target tracking and real-time prediction of objects. Target detection locates and classifies the targets in the field of view through detection algorithm, target tracking is to realize the continuous acquisition of target position information, and trajectory prediction aims to know the motion state and information of moving targets at subsequent time in advance, so as to interact with the targets.

[0003] In the research process, the following problems are found:

[0004] (1) The traditional moving object perception tracking method has too few parameters of the motion equation established for the target, the modeling is too rough, and the prediction accuracy is low;

[0005] (2) The pre-defined anchor boxes are often used in target detection, which leads to complex detection process, and reduces the detection real-time performance and flexibility;

[0006] (3) The tracker algorithm has high complexity, poor real-time performance, large consumption of computing resources and memory resources;

[0007] (4) The prediction accuracy is poor for the nonlinear time-varying scene. SUMMARY

[0008] In order to solve the problems of the prior art, realize the self-perception motion estimation and tracking of flying objects, and improve the tracking efficiency and accuracy, the present application adopts the following technical solutions:

[0009] The object motion estimation and tracking method based on adaptive Kalman filtering comprises the following steps:

[0010] Step 1: kinematic analysis of the object, and construction of the motion equation of the object;

[0011] Step 2: using the target detection FCOS (Fully Convolutional One-stage Object Detector) algorithm to extract and match the features of the moving object, and performing boundary box regression and center degree prediction on each scale to complete the target detection of the moving object;

[0012] Step 3: Based on the KCF (Kernelized Correlation Filter) tracker, the target detection result of the moving object is matched in the target area, the initial detection boundary is taken as the target detection result of the moving object, the features of the region are extracted, the position with the maximum response is found as the tracked target moving object, and the moving object is tracked in real time;

[0013] Step 4: Obtain the state vector of the target moving object; convert the center of mass of the moving object from image coordinates to actual three-dimensional coordinates by using the internal parameters of the depth camera, obtain the position, velocity and acceleration information of the target moving object, and obtain the angular velocity by using the micro gyroscope on the moving object;

[0014] Step 5: Input the state vector of the target moving object into the adaptive Kalman filter, solve the motion equation of the moving object, output the predicted motion information of the target moving object, and estimate and optimize the motion trajectory of the object based on the same, so as to realize the position, velocity and acceleration motion modeling and prediction of the flying target;

[0015] Step 6: Obtain the optical flow information by the PWC-Net (Pyramidal, Warping, and Cost Volume Net) dense optical flow method, input the optical flow information into the KCF tracker, realize autonomous field of view dynamic maintenance, process the target field of view loss problem caused by occlusion, rapid motion or target change, and solve the target moving object loss by optical flow consistency.

[0016] Further, the object motion equation constructed in step 1 is a nonlinear differential equation of the object:

[0017]

[0018] F g = mg

[0019]

[0020] F m = S (ω x v)

[0021] Wherein, m represents the mass of the object, r represents the position vector of the object, t represents time, represents the second order derivative of the position vector with respect to time, F g represents the gravity received by the object, F d represents the air resistance received by the object, F m represents the Magnus force, g represents the gravity acceleration vector, C drepresents the drag coefficient, p represents the air density, A represents the resistance area of the object, v represents the velocity vector of the object, S represents a constant related to the surface roughness of the object and the air density, w represents the angular velocity vector of the object, v represents the velocity vector of the object, the above force is substituted into Newton's second law, the acceleration a is decomposed into position and velocity differential equations, and a nonlinear differential equation of the moving object is obtained:

[0022]

[0023] That is, the corresponding motion equation is obtained:

[0024] a=g+a drag +a magnus

[0025] a drag =-c d ||v||v

[0026] a magnus =c m (ωxv)

[0027] wherein a drag represents the air resistance equivalent acceleration, a magnus represents the Magnus effect equivalent acceleration, represents the air resistance coefficient, represents the Magnus effect coefficient, considering the above three forces, the system has nonlinearity and time variability.

[0028] Further, in step 2, the target detection FCOS algorithm is used to directly predict the target bounding box at each pixel position of the image, first input the target moving object image and feature extraction, the input image is extracted through the pyramid network (FPN) to obtain the feature map F, in the mentioned scene, only the target bounding box needs to be predicted:

[0029] (l,t,r,b)=f reg (F)

[0030] wherein l represents the left distance, t represents the upper distance, r represents the right distance, and b represents the lower distance, f reg represents the convolution layer of the regression branch.

[0031] Further, in step 2, in order to solve the problem of image boundary blur of the target flying ball, the centrality C is used to measure the distance between the predicted point and the center of the moving object:

[0032]

[0033] Wherein, min(, ) represents the minimum value operation, max(, ) represents the maximum value operation, the value of the center degree is between 0 to 1, the higher the center degree, the closer the prediction point to the target center, the regression prediction is carried out for each pixel position of the feature map, the high confidence prediction bounding box is reserved through threshold screening, and the final detection result y is obtained by de-duplication.

[0034] Further, in step 3, the KCF tracker aims to find a filter f, so that the input image patch x i The correlation of the detection result y of the target moving object is maximum, and the objective function is represented as:

[0035]

[0036] Wherein, <f,x i > represents the convolution result of the filter f and the input image x i , y i represents the detection result of the i-th target moving object, and λ represents a regularization parameter, the KCF tracker utilizes Fourier transform to convert the convolution operation into the frequency domain for accelerating the calculation, and the objective function is converted as:

[0037]

[0038] Wherein, And Respectively represent the filter f ball , the detection result y i And the image patch x i The Fourier transform of the kernel matrix, Indicates element multiplication, and the solution of the filter is obtained:

[0039]

[0040] The Gaussian kernel function is selected:

[0041]

[0042] Through Fourier transform, the representation of the kernel matrix in the frequency domain is:

[0043]

[0044] Wherein, x represents the feature vector of the image patch used for tracking in the current frame, z represents the template feature vector of the target model, and σ represents a bandwidth parameter, Indicates Fourier transform;

[0045] In each frame, the KCF tracker updates the filter, and the target position and the new image patch of the current frame are used to update the model, and the update formula is:

[0046]

[0047] Where η represents the learning rate. This indicates the filter for the previous frame. The filter representing the current frame is obtained by solving the frequency domain representation of the convolution operation, and the position of the moving object is calculated using the following formula:

[0048]

[0049] in, This represents the inverse Fourier transform; the location of the maximum value in the response graph is the position of the moving object, and the corresponding moving object target bounding box is output. r ,t r ,r r ,b r ).

[0050] Further, in step 4, a depth image is acquired, and the depth data of the moving object is extracted from it. The centroid of the object is calculated based on the average position of the object's pixels. The centroid is converted from image coordinates to actual three-dimensional coordinates using the camera's intrinsic parameters. The position data of the object is continuously sampled over a period of time, and the speed is calculated based on the rate of change of position.

[0051]

[0052] Calculate acceleration based on the rate of change of velocity:

[0053]

[0054] Where r represents the position vector of the moving target, v represents the velocity vector of the moving target, a represents the acceleration vector of the moving target, k represents the current state, and Δt represents the discrete time step.

[0055] Furthermore, in step 5, the trajectory of the moving target object is updated, and its discrete motion equation is as follows:

[0056]

[0057] The target motion bitmap position, velocity, and acceleration information obtained in step 4 are used as inputs to the adaptive Kalman filter, and the initial state vector x0 = [r0, v0] is set. T Where r0 represents the initial position, v0 represents the initial velocity, the initial control input u0 = a0, a0 represents the initial acceleration, the discrete time step is Δt, and the state transition matrix F and control matrix B are:

[0058]

[0059] Predict the state and covariance at the next time step using the state transition matrix and control matrix, and predict the state vector x. k+1∣k The covariance matrix P k+1∣k for:

[0060] x k+1∣k =Fx k +Bu k

[0061] P k+1∣k =FP k F T +Q k

[0062] Among them, u k =a k Indicates control input, Q k It is the process noise covariance matrix, initially set to the identity matrix Q0 = I. 6×6 The covariance matrix is ​​initially set to the identity matrix P0 = I. 6×6 ;

[0063] Calculate the Kalman gain K by combining the observed values, the updated state of the Kalman gain, and the covariance. k :

[0064] K k =P k+1∣k H T HP k+1∣k H T +R k ) -1

[0065] Update state vector x k+1 The covariance matrix P k+1 :

[0066] y k =z k -Hx k+1∣k

[0067] x k+1 =x k+1∣k +K k y k

[0068] P k+1 =(IK k H)P k+1∣k

[0069] Among them, z k Let H represent the observed values, and let H represent the observation matrix, based on the observed residuals y. k Dynamically adjust process noise covariance matrix Q k and the observation noise covariance matrix R kAdapting to changes in the system:

[0070]

[0071] Where α represents the equilibrium parameter;

[0072] Then, based on the equation of motion of the object in step 1, for u k+1 Update:

[0073]

[0074] Where g represents the gravitational acceleration vector, c d V represents the drag coefficient. k+1 Let S represent the updated velocity vector of the moving object, S represent a constant related to the object's surface roughness and air density, m represent the object's mass, and ω represent the velocity vector of the moving object. k+1 This represents the updated angular velocity vector of the moving object;

[0075] Based on the corrected Kalman gain K k Observation residual y k Update the predicted subsequent state vector x and control input u:

[0076]

[0077] The updated initial vector is the state vector. N represents the discrete time until the end of the motion state, n = 1, 2, 3...N.

[0078] Furthermore, in step 6, pyramid features are extracted from the two images I1 and I2 of the detected moving target object using a CNN:

[0079]

[0080] in, and This represents the feature map at layer l, where L represents the pyramid level, ranging from high resolution to low resolution. At each pyramid level l, the optical flow w estimated from the previous level is used. l+1 Features of the current level Twisting:

[0081]

[0082] Align the features of the second image with the features of the first image to reduce the displacement difference between the images, where x represents the pixel coordinates in the image. After warping, calculate the cost volume:

[0083]

[0084] Where d represents the displacement assumption, the matching cost under different displacement assumptions is represented by comparing the similarity between the distorted features and the features of the first image. A small convolutional network is used to process the cost volume, and the optical flow estimation is refined layer by layer. The optical flow estimation is expressed as:

[0085] w l =ConvNet(C l ,w l+1 )

[0086] Among them, w l This represents the optical flow estimation at layer l, w l+1 This represents the optical flow estimation result from the previous layer, which is further refined through a context network:

[0087]

[0088] Optical flow information is input into the KCF tracker. When target loss occurs due to occlusion, rapid movement, or target change, strategies such as optical flow consistency detection, re-detection mechanism, multi-frame backtracking, and multi-feature fusion are used to handle the target loss situation.

[0089] An object motion estimation and tracking device based on adaptive Kalman filtering includes a kinematic equation construction module, a target detection module, a real-time tracking module, a state acquisition module, a prediction module, and an optical flow information generation module.

[0090] The kinematic equation construction module performs kinematic analysis on the object and constructs the object's equation of motion.

[0091] The object detection module uses the FCOS (Fully Convolutional One-stage Object Detector) algorithm to extract and match features of moving objects, and performs bounding box regression and centrality prediction at each scale to complete the object detection of moving objects.

[0092] The real-time tracking module matches the target region based on the KCF (Kernelized Correlation Filter) tracker, uses the target detection result of the moving object as the initial detection boundary, extracts the features of the region, and finds the position with the largest response as the target moving object to be tracked, so as to track the moving object in real time.

[0093] The state acquisition module is used to acquire the state vector of the target moving object; to convert the centroid of the moving object from image coordinates to actual three-dimensional coordinates using the intrinsic parameters of the depth camera, to obtain the position, velocity and acceleration information of the target moving object, and to obtain the angular velocity through the micro gyroscope on the moving object.

[0094] The prediction module inputs the state vector of the target moving object into an adaptive Kalman filter, solves the motion equation of the moving object, outputs the predicted motion information of the target moving object, and estimates and optimizes the motion trajectory of the object based on this, thereby realizing the modeling and prediction of the position, velocity and acceleration motion of the flying target.

[0095] The optical flow information generation module obtains optical flow information through the PWC-Net (Pyramidal, Warping, and Cost VolumeNet, based on feature pyramid extraction, optical flow mapping, and matching correlation cost measurement) dense optical flow method, and inputs it into the KCF tracker to achieve autonomous dynamic field of view maintenance. It also addresses the problem of target field of view loss caused by occlusion, rapid movement, or target changes. The obtained optical flow information is used to solve the problem of target moving object loss through optical flow consistency.

[0096] The advantages and beneficial effects of this invention are as follows:

[0097] This invention solves the motion equations of a flying sphere using an adaptive Kalman filter algorithm, addresses the dynamic tracking problem based on convolutional features and correlation filtering, and resolves the autonomous field-of-view loss issue using optical flow, thus achieving self-sensory motion estimation and tracking of flying objects. Experiments show that this invention can perform sensing and tracking using the FCOS algorithm and KCF tracker, predict the target flying sphere using the adaptive Kalman filter algorithm, and reduce the probability of target loss using the PWC-Net algorithm, achieving fast and accurate tracking of the flying sphere. Attached Figure Description

[0098] Figure 1 This is a flowchart of the method in an embodiment of the present invention.

[0099] Figure 2 This is a force analysis diagram of the flying sphere target in an embodiment of the present invention.

[0100] Figure 3 This is a schematic diagram of the bounding box regression and centrality prediction results in an embodiment of the present invention.

[0101] Figure 4 This is a schematic diagram of the real-time tracking results of the flying sphere in an embodiment of the present invention.

[0102] Figure 5 This is a schematic diagram illustrating the trajectory prediction method for the self-sensing motion estimation and tracking method of a flying sphere in an embodiment of the present invention.

[0103] Figure 6a This is one of the optical flow information diagrams of the flying sphere in the embodiments of the present invention.

[0104] Figure 6b This is the second optical flow information diagram of the flying sphere in an embodiment of the present invention.

[0105] Figure 6c This is the third optical flow information diagram of the flying sphere in this embodiment of the invention.

[0106] Figure 7 This is a schematic diagram of the device in an embodiment of the present invention. Detailed Implementation

[0107] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0108] like Figure 1 As shown, the object motion estimation and tracking method based on adaptive Kalman filtering includes the following steps:

[0109] Step 1: Perform kinematic analysis based on the wind resistance, rotation, and Magnus effect of the flying sphere to accurately establish the equation of motion of the flying sphere.

[0110] A kinematic analysis and design are performed on the wind resistance, rotation, and Magnus effect involved in the flight of the sphere. The sphere will rotate during its motion. To simplify the model, the sphere is approximated as a six-degree-of-freedom rigid body, subjected to gravity F. g air resistance F d And Magnus force F m Considering the action of these three forces, the system exhibits nonlinearity and time variability, allowing for the establishment of nonlinear differential equations and force analysis as follows: Figure 2 As shown.

[0111]

[0112] Where m is the mass of the sphere, and r is the position vector. The three forces are detailed below:

[0113] F g =mg

[0114]

[0115] F m =S(ω×v)

[0116] Where g is the gravitational acceleration vector, C dHere, ρ is the drag coefficient, ρ is the air density, A is the frontal area of ​​the sphere, and v is the velocity vector of the sphere; the Magnus force is F. m S is a constant related to the surface roughness and air density of the sphere, ω is the angular velocity vector of the sphere, and v is the velocity vector of the sphere. Substituting the above forces into Newton's second law, and decomposing the acceleration a into differential equations of position and velocity, we obtain the nonlinear differential equations of the flying sphere:

[0117]

[0118] That is, the corresponding equation of motion is obtained:

[0119] a=g+a drag +a magnus

[0120] Where r(t) is position, v(t) is velocity, a is acceleration, g is gravitational acceleration, and a drag It is the equivalent acceleration due to air resistance, a magnus The formulas for calculating the Magnus effect equivalent acceleration, the air drag equivalent acceleration, and the Magnus effect equivalent acceleration are as follows:

[0121] a drag =-c d ∥v∥v

[0122] a magnus =c m (ω×v)

[0123] in, It is the air drag coefficient. This is the Magnus effect coefficient. Considering the three forces mentioned above, the system exhibits nonlinearity and time variability.

[0124] Step 2: Use the FCOS algorithm to extract convolutional features and perform feature matching to achieve flying sphere target detection.

[0125] The FCOS algorithm is used to extract convolutional features and perform feature matching. Bounding box regression and centrality prediction are then performed directly at each scale to achieve flying sphere target detection. The results of bounding box regression and centrality prediction are as follows: Figure 3 As shown.

[0126] Specifically, the FCOS algorithm is used to predict the target bounding box directly at each pixel location in the image. First, the target flying sphere image and feature extraction are input. The input image is processed by a pyramid network (FPN) to extract the feature map F. In the scenario mentioned, only the target bounding box needs to be predicted.

[0127] (l,t,r,b)=f reg (F ball )

[0128] Where l represents the left distance, t represents the top distance, r represents the right distance, b represents the bottom distance, and f reg This is a convolutional layer in the regression branch. To address the issue of blurred boundaries in images of a flying sphere, centrality C is used. ball Measuring the distance of the prediction point from the center of the flying sphere:

[0129]

[0130] The centrality value ranges from 0 to 1; a higher centrality indicates that the predicted point is closer to the target center. Regression prediction is performed on each pixel location in the feature map. High-confidence predicted boxes are retained through thresholding, and overlapping boxes are removed using Non-Maximum Suppression (NMS). The final detection result is then y. ball .

[0131] Step 3: Based on the KCF tracker, fast matching is performed in the target area. The detection results of the flying sphere region of the FCOS algorithm are used as the initial detection boundary. The features of this region are extracted, and the position with the largest response is the target flying sphere to be tracked, thus realizing real-time tracking of the flying sphere. The real-time tracking results are as follows: Figure 4 As shown.

[0132] The goal of the KCF tracker is to find a filter f. ball This makes the input image patch x and the target sample y... ball The correlation is the highest. The objective function can be expressed as:

[0133]

[0134] in, <f ball ,x i > represents filter f ball With input image x i The convolution result, Here, λ is the target response, and λ is the regularization parameter. KCF utilizes Fourier transform to convert the convolution operation to the frequency domain. To accelerate computation, the objective function can be transformed into:

[0135]

[0136] in, and They are f ball , and x i Fourier transform of the kernel matrix This represents element-wise multiplication. The solution to the filter can be obtained from this.

[0137]

[0138] Choose the Gaussian kernel function:

[0139]

[0140] The kernel matrix, expressed in the frequency domain via Fourier transform, is as follows: In each frame, KCF updates the filters, using the target location of the current frame and the new image patch to update the model. The update formula is:

[0141]

[0142] Where η is the learning rate. It's the filter from the previous frame. This is the filter for the current frame. The target location can be calculated using the following formula by solving the frequency domain representation of the convolution operation:

[0143]

[0144] in, This represents the inverse Fourier transform. The location of the maximum value in the response graph is the position of the flying sphere, and the corresponding flying sphere target bounding box is output. r ,t r ,r r ,b r ).

[0145] Step 4: Obtain the state vector; reconstruct the target's 3D position, velocity, and acceleration using a depth camera.

[0146] Specifically, the camera's intrinsic parameters are used to convert the center of mass of the flying sphere from image coordinates to actual three-dimensional coordinates, thereby obtaining the target's position, velocity, and acceleration information, and the angular velocity is obtained through a miniature gyroscope on the sphere.

[0147] Depth data of objects is extracted from depth images. The centroid of the object is calculated based on the average position of its pixels. The centroid is then converted from image coordinates to actual 3D coordinates using camera intrinsic parameters. The position data of the object is continuously sampled over a period of time, and the speed is calculated based on the rate of change of position.

[0148]

[0149] Calculate acceleration based on the rate of change of velocity:

[0150]

[0151] Where r is the target's position vector, v is the target's velocity vector, and a is the target's acceleration vector.

[0152] Step 5: Input the target information into the adaptive Kalman filter, solve the nonlinear motion equations of the flying sphere, and output the predicted position, velocity, and acceleration information of the target. Estimate and optimize the object's trajectory, thereby realizing the modeling and prediction of the flying target's position, velocity, and acceleration motion. The output predicted trajectory is as follows: Figure 5 As shown.

[0153] Update the discrete motion equations of the target trajectory:

[0154]

[0155] Using the target position, velocity, and acceleration information from step 4 as input to the adaptive Kalman filter, the initial state vector is set to x0 = [r0, v0]. T Where r0 is the initial position, v0 is the initial velocity, and the initial control input u0 = a0, where a0 is the initial acceleration. The discrete time step is Δt, and the state transition matrix F and control matrix B are:

[0156]

[0157] Predict the state and covariance at the next time step using the state transition matrix and control matrix, and predict the state vector z. k+1∣k The covariance matrix P k+1∣k for

[0158] x k+1∣k =Fx k +Bu k

[0159] P k+1∣k =FP k F T +Q k

[0160] Among them, u k =a k It is the control input, Q k It is the process noise covariance matrix, initially set to the identity matrix Q0 = I. 6×6 The covariance matrix is ​​initially set to the identity matrix P0 = I. 6×6 .

[0161] Calculate the Kalman gain K by combining the observed values, the updated state of the Kalman gain, and the covariance. k :

[0162] K k =P k+1∣k H T HP k+1∣k H T +R k ) -1

[0163] Update state vector x k+1 The covariance matrix P k+1 :

[0164] y k =z k -Hx k+1∣k

[0165] z k+1 =x k+1∣k +K k y k

[0166] P k+1 =(IK k H)P k+1∣k

[0167] Where the observed value is z k H is the observation matrix. Based on the observation residuals y... k Dynamically adjust process noise covariance matrix Q k and the observation noise covariance matrix R k Adapting to changes in the system:

[0168]

[0169] Then, according to the formula listed in step 1, apply it to u. k+1 Update:

[0170]

[0171] Based on the corrected Kalman gain K k Observation residual y k Update the predicted subsequent state vector x and control input u:

[0172]

[0173] Let N be the discrete time until the end of the flight state, then n = 1, 2, 3, ..., N. The updated initial vector is the state vector.

[0174] Step 6: The designed PWC-Net dense optical flow method obtains optical flow information through PWC-Net and inputs it into the KCF tracker to achieve autonomous dynamic field of view maintenance, addressing the problem of target field of view loss caused by occlusion, rapid movement, or target changes. The obtained optical flow information is shown in Figure 6.

[0175] The PWC-Net algorithm is used to address the problem of lost autonomous field of view. Two images, I1 and I2, of the flying spherical target detected in the target detector in step 1 are used to extract pyramid features using a CNN.

[0176]

[0177] in, and The image represents the feature map at layer l, where L represents the pyramid level, ranging from high resolution to low resolution. At each pyramid level l, the optical flow w estimated from previous levels is used. l+1 Features of the current level Twisting:

[0178]

[0179] Align the features of the second image with the features of the first image to reduce the displacement difference between the images, where x is the pixel coordinate in the image. After warping, calculate the cost volume:

[0180]

[0181] Here, d represents the displacement assumption. The matching cost under different displacement assumptions is represented by comparing the similarity between the distorted features and the features of the first image. A small convolutional network is used to process the cost volume, refining the optical flow estimation layer by layer. The optical flow estimation can be expressed as:

[0182] w l =ConvNet(C l ,w l+1 )

[0183] Among them, w l The optical flow estimation is performed at layer l, w l+1 This is the optical flow estimation result from the previous layer. The optical flow estimation result is further refined using a context network:

[0184]

[0185] Optical flow information is input into the KCF tracker. When target loss occurs due to occlusion, rapid movement, or target change, strategies such as optical flow consistency detection, re-detection mechanism, multi-frame backtracking, and multi-feature fusion are used to handle the target loss situation.

[0186] Experimental results show that the method achieves good results in the perception, tracking, and prediction of flying spherical targets.

[0187] like Figure 7 As shown, the device for object motion estimation and tracking based on adaptive Kalman filtering includes a kinematic equation construction module, a target detection module, a real-time tracking module, a state acquisition module, a prediction module, and an optical flow information generation module.

[0188] The kinematic equation construction module performs kinematic analysis on an object and constructs the object's equations of motion.

[0189] The object detection module uses the FCOS (Fully Convolutional One-stage Object Detector) algorithm to extract and match features of moving objects, and performs bounding box regression and centrality prediction at each scale to complete the object detection of moving objects.

[0190] The real-time tracking module matches the target region based on the KCF (Kernelized Correlation Filter) tracker. It uses the target detection result of the moving object as the initial detection boundary, extracts the features of the region, and finds the position with the largest response as the target moving object to be tracked in real time.

[0191] The state acquisition module is used to acquire the state vector of the target moving object; it uses the intrinsic parameters of the depth camera to convert the centroid of the moving object from image coordinates to actual three-dimensional coordinates, obtains the position, velocity and acceleration information of the target moving object, and obtains the angular velocity through the miniature gyroscope on the moving object;

[0192] The prediction module inputs the state vector of the target moving object into an adaptive Kalman filter, solves the motion equation of the moving object, outputs the predicted motion information of the target moving object, and estimates and optimizes the motion trajectory of the object based on this, thereby realizing the modeling and prediction of the position, velocity and acceleration motion of the flying target.

[0193] The optical flow information generation module obtains optical flow information through the PWC-Net (Pyramidal, Warping, and Cost Volume Net, based on feature pyramid extraction, optical flow mapping, and matching correlation cost measurement) dense optical flow method and inputs it into the KCF tracker to achieve autonomous dynamic field of view maintenance. It also addresses the problem of target field of view loss caused by occlusion, rapid movement, or target changes. The obtained optical flow information is used to solve the problem of target moving object loss through optical flow consistency.

[0194] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for object motion estimation and tracking based on adaptive Kalman filtering, characterized in that... Includes the following steps: Step 1: Perform kinematic analysis on the object and construct its equations of motion. The equations of motion are nonlinear differential equations of the moving object. That is, the corresponding equation of motion is obtained: in, Indicates acceleration. Represents the gravitational acceleration vector. Indicates the mass of an object. Indicates the drag coefficient. Indicates density, Represents the area of ​​the object that is obstructed. Represents the velocity vector of an object. This represents a constant related to the surface roughness and density of an object. Represents the angular velocity vector of an object. Represents the velocity vector of an object. Indicates the drag equivalent acceleration, This represents the equivalent acceleration due to the Magnus effect. Indicates the drag coefficient. Indicates the Magnus effect coefficient; Step 2: Use object detection algorithms to extract and match features of moving objects, and perform bounding box regression and centrality prediction at each scale to complete the object detection of moving objects; Step 3: Based on the tracker, match the target area, use the target detection result of the moving object as the initial detection boundary, extract the features of the area, find the position with the largest response as the target moving object to be tracked, so as to track the moving object in real time. Step 4: Obtain the state vector of the target moving object; Step 5: Input the state vector of the target moving object into the adaptive Kalman filter, solve the motion equation of the moving object, output the predicted motion information of the target moving object, and estimate and optimize the motion trajectory of the object based on this; the discrete motion equation for updating the trajectory of the target moving object is as follows: The target motion bitmap position, velocity, and acceleration information obtained in step 4 are used as inputs to the adaptive Kalman filter, and the initial state vector is set. ,in, Indicates the initial position. Indicates initial velocity, initial control input , This represents the initial acceleration, with a discrete time step of . State transition matrix and control matrix for: Predict the state and covariance at the next time step using the state transition matrix and control matrix, and predict the state vector. Covariance Matrix for: in, Indicates control input, It is the process noise covariance matrix, initially set to the identity matrix. The covariance matrix is ​​initially set to the identity matrix. ; Calculate the Kalman gain by combining the observed values ​​with the updated state and covariance of the Kalman gain. : Update state vector Covariance Matrix : in, Represents the observed value. Represents the observation matrix, based on the observation residuals. Dynamic adjustment process noise covariance matrix and observation noise covariance matrix Adapting to changes in the system: in, Indicates the balance parameters; Then, based on the equation of motion of the object from step 1, for Update: in, This represents the updated velocity vector of the moving object. This represents the updated angular velocity vector of the moving object; Based on the corrected Kalman gain Observation residuals Update the predicted subsequent state vector Control input : The updated initial vector is the state vector. , N represents the discrete time until the end of the motion state, n=1,2,3……N; Step 6: Obtain optical flow information using the optical flow method and input it into the tracker. When a target is lost, use the optical flow consistency strategy to handle the target loss situation.

2. The object motion estimation and tracking method based on adaptive Kalman filtering according to claim 1, characterized in that: In step 2, an object detection algorithm is used to directly predict the target bounding box at each pixel location in the image. First, the image of the moving target is input and features are extracted. The input image is processed by a pyramid network to extract feature maps. In the scenario mentioned, only the target bounding box needs to be predicted: in, Indicates the distance to the left. Indicates the distance from the top. Indicates the distance to the right. Indicates the distance to the bottom. This represents the convolutional layer for the regression branch.

3. The object motion estimation and tracking method based on adaptive Kalman filtering according to claim 2, characterized in that: In step 2, centrality is used. Measuring the distance of the prediction point from the center of the moving object: in, This indicates the operation of finding the minimum value. This indicates the operation of taking the maximum value. The higher the centrality, the closer the predicted point is to the target center. Regression prediction is performed on each pixel position of the feature map. Through threshold filtering, high-confidence predicted bounding boxes are retained, and duplicates are removed to obtain the final detection result.

4. The object motion estimation and tracking method based on adaptive Kalman filtering according to claim 1, characterized in that: In step 3, the tracker aims to find a filter. This makes the input image patch The detection results of the target moving object The correlation is the highest, and the objective function is expressed as: in, Represents filter With input image The convolution result, This represents the detection result for the i-th moving target object. The regularization parameter is represented by the Fourier transform parameter. The tracker uses the Fourier transform to convert the convolution operation to the frequency domain, and the objective function is transformed into: in, , and They represent filters respectively. Test results and image patches Fourier transform of the kernel matrix Element-wise multiplication yields the solution for the filter: Choose the Gaussian kernel function: The kernel matrix, expressed in the frequency domain via Fourier transform, is as follows: Where x represents the feature vector of the image patch used for tracking in the current frame, and z represents the template feature vector of the target model. Indicates bandwidth parameter, Indicates Fourier transform; In each frame, the tracker updates the filter, using the target position of the current frame and the new image patch to update the model. The update formula is as follows: in, Indicates the learning rate. This indicates the filter for the previous frame. The filter representing the current frame is obtained by solving the frequency domain representation of the convolution operation, and the position of the moving object is calculated using the following formula: in, This represents the inverse Fourier transform. The position of the maximum value in the response graph is the position of the moving object, and the corresponding bounding box of the moving object is output.

5. The object motion estimation and tracking method based on adaptive Kalman filtering according to claim 1, characterized in that: In step 4, a depth image is acquired, and the depth data of the moving object is extracted from it. The centroid of the object is calculated based on the average position of the object's pixels. The centroid is converted from image coordinates to actual three-dimensional coordinates using the camera's intrinsic parameters. The position data of the object is continuously sampled over a period of time, and the velocity is calculated based on the rate of change of position. Calculate acceleration based on the rate of change of velocity: in, Represents the position vector of the moving target object. This represents the velocity vector of the moving target object. Let k represent the acceleration vector of the moving object, and k represent the current state. This represents the discrete time step.

6. The object motion estimation and tracking method based on adaptive Kalman filtering according to claim 1, characterized in that: In step 6, the two images of the detected moving target object are processed. and Using CNN to extract pyramid features: in, and Indicates the first Feature map of the layer The pyramid levels are represented, from high resolution to low resolution, at each pyramid level. Above, using the optical flow estimated at the previous level. Features of the current level Twisting: Align the features of the second image with the features of the first image, where, Representing pixel coordinates in the image, after distortion, the cost volume is calculated: in, The displacement assumption is represented by comparing the similarity between the distorted features and the features of the first image to indicate the matching cost under different displacement assumptions. A convolutional network is used to process the cost volume, refining the optical flow estimation layer by layer. The optical flow estimation is expressed as: in, Indicates the first Optical flow estimation of the layer, This represents the optical flow estimation result from the previous layer, which is further refined through a context network: Optical flow information is input into the tracker, and when a target is lost, an optical flow consistency strategy is used to handle the target loss situation.

7. An apparatus for object motion estimation and tracking based on adaptive Kalman filtering according to claim 1, comprising a kinematic equation construction module, a target detection module, a real-time tracking module, a state acquisition module, a prediction module, and an optical flow information generation module, characterized in that: The kinematic equation construction module performs kinematic analysis on the object and constructs the object's equation of motion. The target detection module uses a target detection algorithm to extract and match features of moving objects, and performs bounding box regression and centrality prediction at each scale to complete the target detection of moving objects. The real-time tracking module, based on the tracker's matching in the target area, uses the target detection result of the moving object as the initial detection boundary, extracts the features of the area, finds the position with the largest response as the target moving object to be tracked, so as to track the moving object in real time. The state acquisition module is used to acquire the state vector of the target moving object; The prediction module inputs the state vector of the target moving object into an adaptive Kalman filter, solves the motion equation of the moving object, outputs the predicted motion information of the target moving object, and estimates and optimizes the motion trajectory of the object based on this. The optical flow information generation module obtains optical flow information through the optical flow method and inputs it into the tracker. When a target is lost, the optical flow consistency strategy is used to handle the target loss situation.

Images