A method for detecting a moving target with the aid of an IMU in a dynamic situation

Abstract

The application discloses a kind of dynamic situation under the method for detecting moving target assisted by IMU, first by extracting target image feature point matching, obtain feature point pair;Then the relative coordinate conversion is carried out to IMU sensor data, the corresponding basis matrix or homography matrix is solved, and then the corresponding polar line or mapping projection feature point is obtained;After that, the feature point extracted is classified into static background feature point and dynamic target feature point two parts;And according to the corresponding relationship of static background feature point of adjacent two frames of images and motion parameter model, affine transformation matrix is solved, two frames of images are differentiated using interframe difference method and are binarized image and morphological processing, corresponding moving target frame is obtained;Finally, the coincidence degree matching is carried out to moving target frame, and the detection object with coincidence rate greater than set threshold is screened out, i.e. actual moving target in current image.The method can effectively improve the precision and accuracy of moving target recognition under dynamic background.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and computer graphics, and particularly relates to an IMU auxiliary motion target detection method in a dynamic case. BACKGROUND

[0002] Target detection technology in a dynamic background mainly includes two directions of optical flow method and background compensation method. The optical flow method in a dynamic background and the optical flow method in a static background have similar principles, and both are distinguished according to the difference between the optical flow information of the background and the target. In a dynamic background, both the target and the background move, and the optical flow phenomenon appears in the optical flow field, but the speed and direction are different. The principle of the background compensation method is to solve the motion model of the imaging device to represent the image background motion, and to compensate the dynamic background to be a static background. There are two main methods of background compensation through image registration algorithm, which are block motion compensation and global motion compensation. The block motion compensation divides the image uniformly and compares the pixel blocks in the corresponding positions, but this method only considers the translation motion of the background and ignores other transformations such as scaling and rotation, so the applicability is not high. The global motion compensation compares the similarity of the whole image, extracts and matches the features of the adjacent pictures in the video sequence, and solves the affine motion model according to the matching result. The global motion compensation can accurately describe the translation, scaling and rotation of the image. The background compensation method has smaller calculation amount than the optical flow method and has strong adaptability, and has good target detection effect in most cases, but the background compensation accuracy is reduced and the detection accuracy is decreased when facing a complex background. SUMMARY

[0003] The application aims to solve the problem that the existing method has high background complexity, strong environmental variability and large dynamic target proportion in a dynamic case, and the target mark in the detection result is not clear. The application provides an IMU auxiliary motion target detection method in a dynamic case. The method combines the camera motion estimation of the IMU with the improved maximum inter-class variance method to complete the feature point classification task through iterative operation, compares the YOLO target detection information with the motion information of the frame difference detection, and matches the coincidence degree, so as to complete the screening and marking of the actual motion target in the current image, and improve the motion target recognition precision and accuracy in a dynamic background.

[0004] To achieve the above technical purposes, the application adopts the following technical solutions.

[0005] An IMU auxiliary motion target detection method in a dynamic case includes the following steps:

[0006] Step S1, using real-time robust SURF algorithm to extract feature points from adjacent two frames of images, then using BBF search algorithm to match the feature points, and introducing symmetry constraint to obtain robust and accurate feature point pairs;

[0007] Step S11, reading in the video stream, sequentially reading adjacent two frames of images for processing, first using SURF algorithm to extract feature points from adjacent two frames of images, then using BBF search algorithm to find the two nearest feature points in the latter frame; if the second nearest distance divided by the nearest distance is less than a preset threshold, it is determined as a pair of matching points;

[0008] Step S12, using symmetry constraint to remove the mismatched feature point pairs and improve the matching success rate, screening the feature points by bidirectional matching method, for the feature points in the former frame, using forward matching method to find the matching feature points in the current frame, and using reverse matching method to find the matching feature points of the current frame in the former frame, finally obtaining accurate matching feature point pairs;

[0009] Step S2, performing relative coordinate conversion on the IMU sensor data, obtaining the rotation information of the camera by integrating the angular velocity, obtaining the translation information of the camera by twice integrating the acceleration, and solving the corresponding fundamental matrix or homography matrix according to whether there is a position movement between two consecutive frames;

[0010] Step S21, performing relative coordinate conversion on the IMU sensor data according to the relative position relationship between the coordinate system of the camera and the coordinate system of the inertial measurement unit, after performing relative coordinate conversion on the IMU sensor data, the gyroscope and accelerometer contained in the IMU provide the angular velocity and acceleration of the camera respectively, the rotation information of the camera is obtained by integrating the angular velocity, and the translation information of the camera is obtained by twice integrating the acceleration;

[0011] Step S22, calculating the transformation relationship between two consecutive frames according to the obtained camera rotation and translation information, expressing the rotation information between two consecutive frames as:

[0012]

[0013] expressing the translation information between two consecutive frames as:

[0014] T=t k -t k-1 (2)

[0015] In the above formula, R represents the rotation transformation matrix of the former frame image to the current frame image; R k-1 represents the rotation state matrix of the former frame image; R kR represents a rotation state matrix of the current frame image; T represents a translation transformation matrix of the previous frame image to the current frame image; t k-1 R represents a rotation state matrix of the previous frame image; t k R represents a translation state matrix of the current frame image;

[0016] Step S23, solving the corresponding fundamental matrix or homography matrix according to whether there is a position movement of the camera between two continuous image frames; if the translation distance |T| of the previous frame image to the current frame image ≠0, using the IMU sensor data to calculate the initial fundamental matrix, and the calculation formula of the fundamental matrix is as follows:

[0017] F=K -T [T] X RK -1 (3)

[0018] In the above formula, F represents the fundamental matrix corresponding to the previous frame image and the current frame image, R and T are respectively the rotation matrix and the translation matrix of the previous frame to the current frame obtained in step S22, and K is the built-in parameter of the camera;

[0019] If the translation distance |T| of the previous frame image to the current frame image =0, using the IMU sensor data to calculate the initial homography matrix, and the calculation formula of the homography matrix is as follows:

[0020] H=KRK -1 (4)

[0021] In the above formula, H represents the homography matrix corresponding to the previous frame image and the current frame image;

[0022] Step S3, selecting the corresponding fundamental matrix or homography matrix as the input matrix according to whether there is a position movement of the camera between two continuous frames, and then obtaining the homographic mapping projection feature points corresponding to the current frame feature points according to the fundamental matrix and the feature points or the homography matrix and the feature points;

[0023] Step S31, if the translation distance |T| of the previous frame image to the current frame image =0, selecting the corresponding fundamental matrix as the input matrix, and drawing the corresponding epipolar line according to the relationship between the fundamental matrix and the feature points, wherein the epipolar constraint formula is as follows:

[0024]

[0025] In the above formula, p k , p k-1 respectively represent the pixel positions of the corresponding matching feature points of the current frame and the previous frame, F is the fundamental matrix before the two frames, and then:

[0026] l k =Fp k-1 (6)

[0027] In the above formula, l k represents the epipolar line corresponding to the current frame feature point, and has:

[0028] l k : ax+by+c=0 (7)

[0029] In the above formula, x and y are pixel coordinates on l k , a, b and c are the coefficients of the epipolar line l k ;

[0030] Step S32, if the translation distance |T| of the previous frame image to the current frame image ≠0, at this time the epipolar constraint is invalid, the corresponding homography matrix is selected as the input matrix, and the homographic mapping projection feature points corresponding to the current frame feature points are obtained according to the homography matrix and the feature points. The formula for obtaining the homographic mapping projection feature points is as follows:

[0031] p' k =Hp k-1 (8)

[0032] In the above formula, p k-1 , p' k respectively represent the pixel positions of the previous frame feature points and the corresponding current frame homographic projection feature points, and H is the homography matrix before the two frames;

[0033] Step S4, calculate the distance error between each feature point and the corresponding epipolar line or homographic mapping projection feature point, and quantize the distance error of all feature points by histogram. The quantized distance error is input, and the best threshold value is obtained by the maximum inter-class variance method. According to the threshold value, the extracted feature points are classified into two parts: static background feature points and dynamic target feature points;

[0034] Step S41, calculate the distance error between each feature point and the corresponding epipolar line or projection feature point. In the case of input matrix as the fundamental matrix, the distance error is the distance d of the feature point to the corresponding epipolar line:

[0035]

[0036] In the above formula, (u,v), is the coordinate of the feature point, a, b and c are the coefficients of the corresponding epipolar line;

[0037] In the case of input matrix as the homography matrix, the distance error is the distance d between the original feature point and the corresponding projection feature point:

[0038]

[0039] In the above formula, (u' k ,v' k ), (u kv k are the pixel coordinates of the original feature points and the corresponding projected feature points respectively;

[0040] Step S42, quantize the distance error d of each feature point into L intervals from 0 to L, and let the distance error satisfy d i-1 ≤d<d i The feature points belonging to the interval [i-1, i] are d i The total number of feature points N is represented as:

[0041] N=n1+n2+...+ni+...+n k +...+n L (11)

[0042] The probability p i of the number of feature points contained in each interval accounting for the total number is represented as:

[0043]

[0044] Step S43, set a threshold d k , let d min <d k <d max ; and use it to classify the feature points into the static background feature point set C b and the dynamic foreground feature point set C f , wherein C b is composed of all feature points with distance error in the interval [0, k], C f is composed of all feature points with distance error in the interval [k, L], and P b and P f represent the background point set probability and the foreground point set probability respectively, then the probability calculation formula of C b , C f is as follows:

[0045]

[0046] The average distance error of the feature points allocated to C b , C f is respectively:

[0047]

[0048] Step S44, based on the data obtained in step S43, the inter-class variance is represented as:

[0049]

[0050] Since the two means u b and u fThe farther apart the classes are, the greater the inter-class variance is, and the distance error d is traversed between 0 and L i , and the maximum inter-class variance is obtained

[0051]

[0052] The distance error that maximizes the inter-class variance is recorded as the optimal threshold d th ; since the distance error of the static background feature points obtained is smaller than that of the dynamic foreground feature points, when d < d th , it is determined as a static background feature point, and when d > d th , it is determined as a dynamic foreground feature point, and the screened static background feature point set C b and the dynamic foreground feature point set C f are obtained;

[0053] Step S5, the screened static background feature points are combined with the eight-point method to obtain a more accurate fundamental matrix Fn or homography matrix Hn, and steps 3 and 4 are repeated until the classification of all feature points belonging to the set no longer changes, so that the most accurate static background feature points and dynamic target feature points are classified;

[0054] Step S6, an affine transformation matrix is solved according to the corresponding relationship of the static background feature points of the adjacent two frames of images and the motion parameter model, the background of the current frame is compensated, the two frames of images are differentiated and binarized and morphologically processed by using the frame difference method, and the corresponding motion target frame is obtained;

[0055] Step S61, an affine transformation matrix is solved according to the corresponding relationship of the static background feature points of the adjacent two frames of images and the motion parameter model, the motion parameter value is obtained, the current frame is compensated according to the accurate global motion parameter, the bilinear interpolation algorithm is used to interpolate the gray image after background compensation, and the two frames of images are differentiated by using the frame difference method;

[0056] Step S62, the gray image after frame difference is subjected to adaptive threshold binarization processing and Canny edge detection, and the results are morphologically processed to obtain the corresponding motion target frame;

[0057] Step S7, the YOLO network is used to detect all potential objects in the image, and the object target frame detected by YOLO is obtained;

[0058] Step S8, the motion target frame is traversed, the motion target frame whose center point is located in the object target frame detected by YOLO is merged, the complete motion target frame is obtained, and the merged motion target frame information is matched with the target frame information detected by YOLO, and the calculation formula of the coincidence rate is as follows:

[0059]

[0060] In the above formula, λ represents the coincidence rate of the motion target rectangular frame Z1 detected by the inter-frame difference detection and the target rectangular frame Z2 detected by the YOLO detection;

[0061] Step S9: The detection object with the coincidence rate greater than the set threshold value is screened out, that is, the actual motion target in the current image.

[0062] Compared with the prior art, the present application has the following beneficial effects:

[0063] The method of the present application combines the camera motion estimation based on the IMU with the improved maximum inter-class variance method to complete the feature point classification task through iterative operation, and compares the YOLO target detection information with the motion information detected by the inter-frame difference detection and matches the coincidence degree, thereby completing the screening and marking the actual motion target in the current image, and achieving the effect of improving the motion target recognition precision and accuracy in the dynamic background. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present disclosure, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0065] Figure 1 is a flowchart of the IMU-assisted motion target detection method in a dynamic situation according to the present application;

[0066] Figure 2 is an example diagram of feature point matching in the embodiments of the present application;

[0067] Figure 3 is an epipolar drawing and a feature point homography mapping drawing in the embodiments of the present application;

[0068] Figure 4 is a coincidence degree matching flowchart of the motion target frame information and the target frame information detected by the YOLO in the embodiments of the present application;

[0069] Figure 5 is an example diagram of the motion target detection process in the embodiments of the present application. DETAILED DESCRIPTION

[0070] For the convenience of those skilled in the art to understand and implement the present application, the steps of the method of the present application are described in detail below, and it should be understood that these examples are only used to illustrate the present application and not to limit the scope of the present application. In addition, it should be understood that those skilled in the art can make various modifications or modifications to the present application after reading the content taught by the present application, and these equivalent forms also fall within the scope defined by the claims attached to the present application.

[0071] Embodiments

[0072] As shown in Figure 1 , the present application discloses a dynamic IMU-assisted moving target detection method, comprising the following steps:

[0073] Step S1, using a real-time robust SURF algorithm to extract feature points from adjacent two frames of images, then using a BBF search algorithm to match the feature points, and introducing a symmetry constraint to obtain robust and accurate feature point pairs;

[0074] Step S11, read in the video stream, read adjacent two frames of images in turn for processing, first use the SURF algorithm to extract feature points from adjacent two frames of images, then use the BBF search algorithm to find the two nearest feature points in the latter frame; If the next nearest distance divided by the nearest distance is less than a predetermined threshold, it is determined as a pair of matching points;

[0075] Step S12, using symmetry constraint to remove false matching feature point pairs and improve matching success rate, screening the feature points by bidirectional matching method, for the feature points in the previous frame, using the forward matching method to find the matching feature points in the current frame, and using the reverse matching method to find the matching feature points of the current frame in the previous frame, as shown in Figure 2 , the final accurate matching feature point pairs are obtained;

[0076] Step S2, performing relative coordinate conversion on the IMU sensor data, obtaining the rotation information of the camera by integrating the angular velocity, obtaining the translation information of the camera by twice integrating the acceleration, and solving the corresponding fundamental matrix or homography matrix according to whether there is a position movement between the camera in two consecutive frames;

[0077] Step S21, performing relative coordinate conversion on the IMU sensor data according to the relative position relationship between the coordinate system of the camera and the coordinate system of the inertial measurement unit. After the relative coordinate conversion of the IMU sensor data, the gyroscope and accelerometer contained in the IMU provide the angular velocity and acceleration of the camera, respectively. The rotation information of the camera is obtained by integrating the angular velocity, and the translation information of the camera is obtained by twice integrating the acceleration.

[0078] Step S22, the transformation relationship between two consecutive frames is calculated according to the obtained camera rotation and translation information, and the rotation information between two consecutive frames is expressed by a formula as follows:

[0079]

[0080] The translation information between two consecutive frames is expressed by a formula as follows:

[0081] T=t k -t k-1 (2)

[0082] In the above formula, R represents a rotation transformation matrix of a previous frame image to a current frame image; R k-1 represents a rotation state matrix of the previous frame image; R k represents a rotation state matrix of the current frame image; T represents a translation transformation matrix of the previous frame image to the current frame image; t k-1 represents a translation state matrix of the previous frame image; t k represents a translation state matrix of the current frame image;

[0083] Step S23, whether there is a position movement between two consecutive image frames is determined, and a corresponding fundamental matrix or homography matrix is solved; if a translation distance |T| of the previous frame image to the current frame image ≠ 0, an initial fundamental matrix is calculated using IMU sensor data, and a calculation formula of the fundamental matrix is as follows:

[0084] F=K -T [T] X RK -1 (3)

[0085] In the above formula, F represents a fundamental matrix corresponding to the previous frame image and the current frame image, R and T are respectively a rotation matrix and a translation matrix of the previous frame to the current frame obtained in step S22, and K is a built-in parameter of the camera;

[0086] If the translation distance |T| of the previous frame image to the current frame image = 0, an initial homography matrix is calculated using IMU sensor data, and a calculation formula of the homography matrix is as follows:

[0087] H=KRK -1 (4)

[0088] In the above formula, H represents a homography matrix corresponding to the previous frame image and the current frame image;

[0089] Step S3, select the corresponding fundamental matrix or homography matrix as the input matrix according to whether there is a position movement between two continuous frames of the camera, then draw the epipolar line according to the fundamental matrix and the feature points or obtain the homographic mapping projection feature points corresponding to the current frame feature points according to the homography matrix and the feature points, for example Figure 3 , Figure 3 , Figure 3 , Figure 3 , Figure 3 , ,

[0090] Step S31, if the translation distance |T| = 0 from the previous frame image to the current frame image, select the corresponding fundamental matrix as the input matrix, and draw the corresponding epipolar line according to the relationship between the fundamental matrix and the feature points, wherein the epipolar constraint formula is as follows:

[0091]

[0092] In the above formula, p k and p k-1 represent the pixel positions of the corresponding matching feature points of the current frame and the previous frame, F is the fundamental matrix before the two frames, and then:

[0093] l k = Fp k-1 (6)

[0094] In the above formula, l k represents the epipolar line corresponding to the current frame feature point, and then:

[0095] l k : ax+by+c=0 (7)

[0096] In the above formula, x and y are pixel coordinates on l k , and a, b and c are coefficients of the epipolar line l k ;

[0097] Step S32, if the translation distance |T|≠0 from the previous frame image to the current frame image, the epipolar constraint is invalid at this time, select the corresponding homography matrix as the input matrix, and obtain the homographic mapping projection feature points corresponding to the current frame feature points according to the homography matrix and the feature points, and the obtaining formula of the homographic mapping projection feature points is as follows:

[0098] p' k = Hp k-1 (8)

[0099] In the above formula, p k-1 and p' kThey represent the pixel positions of the feature points of the previous frame and the corresponding feature points of the current frame homography projection, respectively, and H is the homography matrix before the two frames;

[0100] Step S4: Calculate the distance error between each feature point and the corresponding epipolar line or homography projection feature point, perform histogram quantization on the distance errors of all feature points, use the quantized distance error as input, and use the maximum inter-class variance method to find the optimal threshold. Based on the threshold, classify the extracted feature points into two parts: static background feature points and dynamic target feature points.

[0101] Step S41: Calculate the distance error between each feature point and the corresponding epipolar line or projected feature point. When the input matrix is ​​the basic matrix, the distance error is the distance d from the feature point to the corresponding epipolar line:

[0102]

[0103] In the above formula, (u, v) is the coordinate of the feature point, a, b and c are the coefficients of the corresponding polar line;

[0104] When the input matrix is ​​a homography matrix, the distance error is the distance d between the original feature point and the corresponding projected feature point:

[0105]

[0106] In the above formula, (u' k ,v' k )、(u k ,v k ) are the pixel coordinates of the original feature point and the corresponding projected feature point respectively;

[0107] Step S42: quantize the distance error d of each feature point into L intervals from 0 to L. Assume that the distance error satisfies d i-1 ≤d <d i The feature points belong to the interval [i-1,i], and the number of corresponding feature points is n i The total number of feature points N is expressed as:

[0108] N=n1+n2+...+n k +...+n L (11)

[0109] The probability p that the number of feature points contained in each interval accounts for the total number i Expressed as:

[0110]

[0111] Step S43: Set a threshold d k , let d min <d k<d max ; and use it to classify feature points into static background feature point set C b and dynamic foreground feature point set C f , where C b is composed of all feature points whose distance errors are in the interval [0, k], and C f is composed of all feature points whose distance errors are in the interval [k, L], denoted as P b and P f respectively, the background point set probability and the foreground point set probability, then the probability of C b , C f is calculated as follows:

[0112]

[0113] The average distance error of feature points assigned to C b , C f point set is respectively:

[0114]

[0115] Step S44, based on the data obtained in step S43, the inter-class variance is expressed as:

[0116]

[0117] Since the two means u b and u f are farther away from each other, the inter-class variance is greater. Traverse the distance error d i from 0 to L, and get the maximum inter-class variance

[0118]

[0119] Let the distance error with the maximum inter-class variance be the optimal threshold d th ; since the distance error of the static background feature points obtained is smaller than that of the dynamic foreground feature points, when d < d th , it is determined as a static background feature point, and when d > d th , it is determined as a dynamic foreground feature point, and the screened static background feature point set C b and the dynamic foreground feature point set C f are obtained;

[0120] Step S5, combine the screened static background feature points with the eight-point method to obtain a more accurate fundamental matrix Fn or homography matrix Hn, repeat steps 3 and 4 until the classification of all feature points belonging to the set no longer changes, thereby obtaining the most accurate static background feature point and dynamic target feature point classification, for example, the feature point classification is as followsFigure 5 In (b), the current frame is the original frame, Figure 5 In (a), the current frame is the original frame, Figure 5 In (a), only one car on the road is moving, and the other vehicles in the parking lot are stationary;

[0121] Step S6, affine transformation matrix is solved according to the corresponding relationship of static background feature points and motion parameter model of adjacent two frames of images, background compensation is performed on the current frame, binary image and morphological processing are performed on the two frames of images by using inter-frame difference method, and corresponding motion target frame is obtained;

[0122] Step S61, affine transformation matrix is solved according to the corresponding relationship of static background feature points and motion parameter model of adjacent two frames of images, motion parameter value is obtained, background compensation is performed on the current frame according to accurate global motion parameter, interpolation is performed on the gray image after background compensation by using bilinear interpolation algorithm, and inter-frame difference processing is performed on the two frames of images, as shown in Figure 5 In (c), the inter-frame difference effect diagram is shown, Figure 5 In (d), the binary processing effect diagram is shown, Figure 5 In (e), the morphological processing effect diagram is shown; Figure 5 In (e), the morphological processing effect diagram is shown;

[0123] Step S62, adaptive threshold binary processing and Canny edge detection are performed on the gray image after inter-frame difference, and morphological processing is performed on the results of the two, and corresponding motion target frame is obtained;

[0124] Step S7, YOLO network is used to detect all potential objects in the image, as shown in Figure 5 In (f), the object target frame detected by YOLO is obtained;

[0125] Step S8, the motion target frame is traversed, the motion target frame with the center point located in the object target frame detected by YOLO is merged to obtain a complete motion target frame, and the merged motion target frame information is matched with the target frame information detected by YOLO, and the matching process is as shown in Figure 4 The calculation formula of the coincidence rate is as follows:

[0126]

[0127] In the above formula, λ represents the coincidence rate of the motion target rectangular frame Z1 detected by inter-frame difference and the target rectangular frame Z2 detected by YOLO;

[0128] Step S9, the detection object with the coincidence rate greater than the set threshold value is screened out, which is the actual motion target in the current image, and the motion target detection result in the embodiment is as shown in Figure 5 In (g), the motion target detection result is shown, Figure 5The middle (h) is a true value of the moving target.

[0129] In summary, the method of the present application combines the camera motion estimation based on IMU with the improved maximum inter-class variance method to complete the feature point classification task through iterative operation, and compares the YOLO target detection information with the motion information detected by frame difference detection and matches the coincidence degree, and then completes the screening and marking of the actual moving target in the current image, so as to improve the accuracy and accuracy of moving target recognition in a dynamic background.

[0130] The above is only a preferred embodiment of the present application, not other forms of the present application, any skilled in the art can use the above disclosed technical content to change or modify as equivalent embodiments of equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiments without departing from the technical solution content of the present application, according to the technical essence of the present application, still belongs to the protection scope of the technical solution of the present application.

Claims

1. A method for detecting moving targets in a dynamic situation using an IMU, characterized in that: The following steps are involved: Step S1: Use the real-time robust SURF algorithm to extract feature points from two adjacent frames of images, then use the BBF search algorithm to match the feature points, and introduce symmetry constraints to obtain robust and accurate feature point pairs; Step S2: Perform relative coordinate transformation on the IMU sensor data, obtain the camera's rotation information by integrating the angular velocity, obtain the camera's translation information by quadratically integrating the acceleration, and solve the corresponding fundamental matrix or homography matrix based on whether the camera moves between two consecutive frames. Step S3: Select the corresponding fundamental matrix or homography matrix as the input matrix according to whether the camera moves between two consecutive frames, and then draw epipolar lines based on the fundamental matrix and the feature points, or obtain the homography projection feature points corresponding to the feature points of the current frame based on the homography matrix and the feature points; Step S4: Calculate the distance error between each feature point and the corresponding epipolar line or homography projection feature point, perform histogram quantization on the distance errors of all feature points, use the quantized distance error as input, and use the maximum inter-class variance method to find the optimal threshold. Based on the threshold, classify the extracted feature points into two parts: static background feature points and dynamic target feature points. Step S5: Combine the selected static background feature points with the eight-point method to solve a more accurate basic matrix Fn or homography matrix Hn. Repeat steps S3 and S4 until the classification of the set to which all feature points belong no longer changes, thereby obtaining the most accurate classification of static background feature points and dynamic target feature points. Step S6: solving the affine transformation matrix based on the correspondence between the static background feature points of the two adjacent frames and the motion parameter model, performing background compensation on the current frame, performing difference between the two frames using the inter-frame difference method, and performing binarization and morphological processing to obtain the corresponding moving target frame; Step S7: Use the YOLO network to detect all potential objects in the image and obtain the object target frame detected by YOLO; Step S8: traverse the moving target frame, merge the moving target frame whose center point is located in the object target frame detected by YOLO, obtain a complete moving target frame, and match the merged moving target frame information with the target frame information detected by YOLO; Step S9: Screen out the detection objects whose overlap rate is greater than the set threshold, which are the actual moving targets in the current image.

2. The IMU-assisted moving target detection method under dynamic conditions according to claim 1, characterized in that: The specific contents of step 1 are as follows: Step S11: read in the video stream and sequentially read two adjacent frames for processing. First, the SURF algorithm is used to extract feature points from the two adjacent frames. Then, the BBF search algorithm is used to find the two closest feature points in the next frame. If the next closest distance divided by the closest distance is less than a preset threshold, the points are determined to be a pair of matching points. Step S12: Use symmetric constraints to remove mismatched feature point pairs and improve the matching success rate. The feature points are screened through the two-way matching method. For the feature points in the previous frame, the forward matching method is used to find the matching feature points in the current frame. Then, the reverse matching method is used to find the matching feature points of the current frame in the previous frame, and finally an accurate matching feature point pair is obtained.

3. The IMU-assisted moving target detection method under dynamic conditions according to claim 1, characterized in that: The specific content of step S2 is as follows: Step S21: Perform relative coordinate transformation on the IMU sensor data based on the relative positional relationship between the camera coordinate system and the inertial measurement unit coordinate system. After the relative coordinate transformation, the gyroscope and accelerometer included in the IMU provide the camera's angular velocity and acceleration, respectively. The camera's rotation information is obtained by integrating the angular velocity, and the camera's translation information is obtained by quadratically integrating the acceleration. Step S22: Calculate the transformation relationship between two consecutive frames based on the obtained camera rotation and translation information, and express the rotation information between two consecutive frames using the formula: The translation information between two consecutive frames is expressed as: T=t k -t k-1 (2) In the above formula, R represents the rotation transformation matrix from the previous frame image to the current frame image; R k-1 Represents the rotation state matrix of the previous frame image; R k represents the rotation state matrix of the current frame image; T represents the translation transformation matrix from the previous frame image to the current frame image; t k-1 Represents the translation state matrix of the previous frame image; t k Represents the translation state matrix of the current frame image; Step S23: solving the corresponding basic matrix or homography matrix according to whether the camera moves between two consecutive image frames; If the translation distance |T| from the previous frame image to the current frame image is ≠ 0, the IMU sensor data is used to calculate the initial basic matrix. The calculation formula of the basic matrix is ​​as follows: F=K -T [T] X RK -1 (3) In the above formula, F represents the basic matrix corresponding to the previous frame image and the current frame image, R and T are the rotation matrix and translation matrix from the previous frame to the current frame obtained in step S22, respectively, and K is the built-in parameter of the camera; If the translation distance |T| from the previous frame image to the current frame image = 0, the IMU sensor data is used to calculate the initial homography matrix. The calculation formula of the homography matrix is ​​as follows: H=KRK -1 (4) In the above formula, H represents the homography matrix corresponding to the previous frame image and the current frame image.

4. The IMU-assisted moving target detection method under dynamic conditions according to claim 1, characterized in that: The specific content of step S3 is as follows: Step S31: If the translation distance |T| from the previous frame image to the current frame image is 0, the corresponding basic matrix is ​​selected as the input matrix, and the corresponding epipolar line is drawn according to the relationship between the basic matrix and the feature points. The epipolar constraint formula is as follows: In the above formula, p k 、p k-1 Represent the pixel positions of the corresponding matching feature points in the current frame and the previous frame respectively, and F is the basic matrix before the two frames, then: l k =Fp k-1 (6) In the above formula, l k Represents the epipolar line corresponding to the feature point of the current frame, and has: l k :ax+by+c=0 (7) In the above formula, x and y are l k The pixel coordinates on the line a, b, and c are the epipolar lines l k The coefficient of Step S32: If the translation distance |T| from the previous frame image to the current frame image is ≠ 0, the epipolar constraint is invalid at this time, and the corresponding homography matrix is ​​selected as the input matrix. The homography projection feature points corresponding to the feature points of the current frame are obtained based on the homography matrix and the feature points. The formula for obtaining the homography projection feature points is as follows: p′ k =Hp k-1 (8) In the above formula, p k-1 、p' k They represent the pixel positions of the feature points of the previous frame and the corresponding homography projection feature points of the current frame, respectively, and H is the homography matrix before the two frames.

5. The IMU-assisted moving target detection method under dynamic conditions according to claim 1, characterized in that: The specific content of step S4 is as follows: Step S41: Calculate the distance error between each feature point and the corresponding epipolar line or projected feature point. When the input matrix is ​​the basic matrix, the distance error is the distance d from the feature point to the corresponding epipolar line: In the above formula, (u, v) is the coordinate of the feature point, a, b and c are the coefficients of the corresponding polar line; When the input matrix is ​​a homography matrix, the distance error is the distance d between the original feature point and the corresponding projected feature point: In the above formula, (u'k,v'k), (u k ,v k ) are the pixel coordinates of the original feature point and the corresponding projected feature point respectively; Step S42: quantize the distance error d of each feature point into L intervals from 0 to L. Assume that the distance error satisfies d i-1 ≤d <d i The feature points belong to the interval [i-1,i], and the number of corresponding feature points is n i The total number of feature points N is expressed as: N=n1+n2+...+n k +...+n L (11) The probability p that the number of feature points contained in each interval accounts for the total number i Expressed as: Step S43: Set a threshold d k , let d min <d k <d max ; and use it to classify feature points into static background feature point set C b and dynamic foreground feature point set C f , where C b It consists of all feature points whose distance error is within the interval [0, k], C f It is composed of all feature points whose distance error is within the interval [k, L], and is represented by P b and P f Represents the probability of background point set and foreground point set, then C b 、C f The probability calculation formula is as follows: Assign to C b 、C f The average distance errors of the feature points of the point set are: Step S44: Based on the data obtained in step S43, the between-class variance Expressed as: Since the two means u b and u f The farther apart they are, the greater the inter-class variance is, and the distance error d is traversed from 0-L. i , get the maximum between-class variance Let the distance error with the largest inter-class variance be recorded as the optimal threshold d th ; Since the distance error of the static background feature points is smaller than the distance error of the dynamic foreground feature points, when d <d th It is judged as a static background feature point when d>d th When it is judged as a dynamic foreground feature point, the filtered static background feature point set C is obtained b and dynamic foreground feature point set C f .

6. The IMU-assisted moving target detection method under dynamic conditions according to claim 1, characterized in that: The specific content of step S6 is as follows: Step S61: Solve the affine transformation matrix based on the correspondence between the static background feature points of two adjacent frames and the motion parameter model to obtain the motion parameter value, perform background compensation on the current frame based on the accurate global motion parameter, interpolate the grayscale image after background compensation using a bilinear interpolation algorithm, and perform difference processing on the two frames using an inter-frame difference method; Step S62: perform adaptive threshold binarization processing and Canny edge detection on the grayscale image after inter-frame difference, and perform morphological processing on the results to obtain the corresponding moving target frame.

7. The IMU-assisted moving target detection method in dynamic conditions according to claim 1, characterized in that: In step S8, the merged moving target frame information is matched with the target frame information detected by YOLO. The calculation formula of the overlap rate is as follows: In the above formula, λ represents the overlap ratio between the moving target rectangular frame Z1 detected by inter-frame difference and the target rectangular frame Z2 detected by YOLO.

Images