IMU (Inertial Measurement Unit)-assisted moving target detection method under dynamic condition

Abstract

The invention discloses an IMU (Inertial Measurement Unit)-assisted moving target detection method under a dynamic condition, which comprises the following steps of: firstly, extracting feature points of a target image for matching to obtain feature point pairs; then, relative coordinate conversion is carried out on IMU sensor data, a corresponding basic matrix or homography matrix is solved, and then corresponding polar lines or mapping projection feature points are obtained; classifying the extracted feature points into two parts, namely static background feature points and dynamic target feature points; an affine transformation matrix is solved according to the corresponding relation of the static background feature points of the two adjacent frames of images and the motion parameter model, an inter-frame difference method is used for carrying out difference on the two frames of images and carrying out image binarization and morphological processing, and a corresponding motion target frame is obtained; and finally, traversing the moving target frame to carry out overlap ratio matching, and screening out a detection object of which the overlap ratio is greater than a set threshold value, namely an actual moving target in the current image. The method provided by the invention can effectively improve the precision and accuracy of moving target identification in a dynamic background.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and computer graphics, and in particular relates to an IMU-assisted moving target detection method under dynamic conditions. Background Art

[0002] The target detection technology under dynamic background is mainly divided into two directions: optical flow method and background compensation method. The principle of the optical flow method under dynamic background is similar to that of the optical flow method under static background. They are also distinguished based on the difference in optical flow information between the background and the target. In the dynamic background, both the target and the background move. In the optical flow field, both will have optical flow phenomenon, but the speed and direction are different. The principle of the background compensation method is to characterize the motion of the image background by solving the motion model of the imaging device, and convert the dynamic background into a static background after motion compensation. There are two main methods for background compensation through image registration algorithm, block-based and global motion compensation. Block-based motion compensation is to divide the image evenly and only compare the pixel blocks at the corresponding position. However, this method only considers the translational motion of the background and ignores other possible transformations such as scaling and rotation. Therefore, it is not very applicable. Global motion compensation is to compare the similarity of the entire image. By extracting and matching features of adjacent pictures in the video sequence, the affine motion model is solved according to the matching results. Global motion compensation can accurately describe various transformations such as translation, scaling and rotation between images. The background compensation method has less computational complexity than the optical flow method and has stronger adaptability. It has better target detection effects in most cases. However, when faced with complex backgrounds, the background compensation accuracy is reduced and the detection accuracy decreases. Summary of the invention

[0003] The purpose of the present invention is to provide an IMU-assisted moving target detection method under dynamic conditions in view of the problems that the existing methods have high background complexity, strong environmental variability, and unclear target markings in detection results when the proportion of dynamic targets is too large. The method completes the feature point classification task based on IMU camera motion estimation and an improved maximum inter-class variance method combined with iterative calculation, and compares and matches the coincidence of YOLO target detection information with the motion information of inter-frame difference detection, thereby completing the screening and marking of the actual moving targets in the current image, thereby achieving the effect of improving the precision and accuracy of moving target recognition under dynamic background.

[0004] In order to achieve the above technical objectives, the present invention adopts the following technical solutions.

[0005] A method for detecting a moving target assisted by an IMU in a dynamic situation comprises the following steps:

[0006] Step S1, using the real-time robust SURF algorithm to extract feature points from two adjacent frames of images, then using the BBF search algorithm to match the feature points, and introducing symmetry constraints to obtain robust and accurate feature point pairs;

[0007] Step S11, read in the video stream, read two adjacent frames of images in sequence for processing, first use the SURF algorithm to extract feature points from the two adjacent frames of images, and then use the BBF search algorithm to find the two closest feature points in the next frame; if the next closest distance divided by the closest distance is less than a preset threshold, it is determined to be a pair of matching points;

[0008] Step S12: using symmetric constraints to remove mismatched feature point pairs and improve the matching success rate, the feature points are screened by a bidirectional matching method, and for the feature points in the previous frame, the forward matching method is used to find the matching feature points in the current frame, and then the reverse matching method is used to find the matching feature points of the current frame in the previous frame, and finally an accurate matching feature point pair is obtained;

[0009] Step S2, performing relative coordinate transformation on the IMU sensor data, obtaining the rotation information of the camera by integrating the angular velocity, obtaining the translation information of the camera by quadratically integrating the acceleration, and solving the corresponding basic matrix or homography matrix according to whether the camera moves between two consecutive frames;

[0010] Step S21, performing relative coordinate transformation on the IMU sensor data according to the relative position relationship between the coordinate system where the camera is located and the coordinate system where the inertial measurement unit is located. After the relative coordinate transformation on the IMU sensor data, the gyroscope and accelerometer included in the IMU respectively provide the angular velocity and acceleration of the camera, and the rotation information of the camera is obtained by integrating the angular velocity, and the translation information of the camera is obtained by quadratically integrating the acceleration;

[0011] Step S22: Calculate the transformation relationship between two consecutive frames according to the obtained camera rotation and translation information, and express the rotation information between two consecutive frames by the formula:

[0012]

[0013] The translation information between two consecutive frames is expressed as:

[0014] T=t k -t k-1 (2)

[0015] In the above formula, R represents the rotation transformation matrix from the previous frame image to the current frame image; R k-1 Represents the rotation state matrix of the previous frame image; R krepresents the rotation state matrix of the current frame image; T represents the translation transformation matrix from the previous frame image to the current frame image; t k-1 Represents the translation state matrix of the previous frame image; t k Represents the translation state matrix of the current frame image;

[0016] Step S23, solve the corresponding basic matrix or homography matrix according to whether the camera moves between two consecutive image frames; if the translation distance |T| from the previous frame image to the current frame image is ≠ 0, use the IMU sensor data to calculate the initial basic matrix, and the calculation formula of the basic matrix is ​​as follows:

[0017] F=K -T [T] X R -1 (3)

[0018] In the above formula, F represents the basic matrix corresponding to the previous frame image and the current frame image, R and T are respectively the rotation matrix and translation matrix from the previous frame to the current frame obtained in step S22, and K is the built-in parameter of the camera;

[0019] If the translation distance |T| from the previous frame image to the current frame image = 0, the IMU sensor data is used to calculate the initial homography matrix. The calculation formula of the homography matrix is ​​as follows:

[0020] H=KRK -1 (4)

[0021] In the above formula, H represents the homography matrix corresponding to the previous frame image and the current frame image;

[0022] Step S3, selecting a corresponding basic matrix or homography matrix as an input matrix according to whether the camera moves between two consecutive frames, and then drawing epipolar lines according to the basic matrix and feature points, or obtaining homography mapping projection feature points corresponding to feature points of the current frame according to the homography matrix and feature points;

[0023] Step S31: If the translation distance |T| from the previous frame image to the current frame image is 0, the corresponding basic matrix is ​​selected as the input matrix, and the corresponding epipolar line is drawn according to the relationship between the basic matrix and the feature points, wherein the epipolar constraint formula is as follows:

[0024]

[0025] In the above formula, p k 、p k-1 Respectively represent the pixel positions of the corresponding matching feature points in the current frame and the previous frame, and F is the basic matrix before the two frames, then:

[0026] l k =Fp k-1 (6)

[0027] In the above formula, l k Represents the epipolar line corresponding to the feature point of the current frame, and has:

[0028] l k :ax+by+c=0 (7)

[0029] In the above formula, x and y are l k The pixel coordinates on the , a, b and c are the epipolar lines l k The coefficient of

[0030] Step S32: If the translation distance |T| from the previous frame image to the current frame image is ≠ 0, the epipolar constraint is invalid at this time, and the corresponding homography matrix is ​​selected as the input matrix. The homography mapping projection feature points corresponding to the feature points of the current frame are obtained according to the homography matrix and the feature points. The formula for obtaining the homography mapping projection feature points is as follows:

[0031] p' k =Hp k-1 (8)

[0032] In the above formula, p k-1 、p' k They represent the pixel positions of the feature points of the previous frame and the feature points of the corresponding homography projection of the current frame, respectively, and H is the homography matrix before two frames;

[0033] Step S4, calculating the distance error between each feature point and the corresponding epipolar line or homography projection feature point, performing histogram quantization on the distance errors of all feature points, taking the quantized distance error as input, and obtaining the optimal threshold by the maximum inter-class variance method, and classifying the extracted feature points into two parts: static background feature points and dynamic target feature points according to the threshold;

[0034] Step S41, calculate the distance error between each feature point and the corresponding epipolar line or the projected feature point. When the input matrix is ​​the basic matrix, the distance error is the distance d from the feature point to the corresponding epipolar line:

[0035]

[0036] In the above formula, (u, v) is the coordinate of the feature point, a, b and c are the coefficients of the corresponding polar line;

[0037] When the input matrix is ​​a homography matrix, the distance error is the distance d between the original feature point and the corresponding projected feature point:

[0038]

[0039] In the above formula, (u' k ,v' k )、(u k,v k ) are the pixel coordinates of the original feature point and the corresponding projected feature point respectively;

[0040] Step S42: quantize the distance error d of each feature point into L intervals from 0 to L. Assume that the distance error satisfies d i-1 ≤d <d i The feature points belong to the interval [i-1,i], and the corresponding number of feature points is n i Then the total number of feature points N is expressed as:

[0041] N=n1+n2+...+n k +...+n L (11)

[0042] The probability p that the number of feature points contained in each interval accounts for the total number i It is expressed as:

[0043]

[0044] Step S43: Set a threshold d k , let d min <d k <d max ; and use it to classify the feature points into a static background feature point set C b and dynamic foreground feature point set C f , where C b It consists of all feature points whose distance error is in the interval [0, k], C f It is composed of all feature points whose distance error is in the interval [k,L], respectively represented by P b and P f represents the probability of background point set and foreground point set, then C b , C f The probability calculation formula is as follows:

[0045]

[0046] Assign to C b , C f The average distance errors of the feature points of the point set are:

[0047]

[0048] Step S44: Based on the data obtained in step S43, the between-class variance It is expressed as:

[0049]

[0050] Since the two means u b and u fThe farther apart they are, the greater the inter-class variance, and the distance error d traversed from 0-L i , get the maximum between-class variance

[0051]

[0052] Let the distance error with the largest inter-class variance be recorded as the optimal threshold d th ; Since the distance error of the static background feature points is smaller than the distance error of the dynamic foreground feature points, when d <d th When d>d th When it is judged as a dynamic foreground feature point, the filtered static background feature point set C is obtained b and dynamic foreground feature point set C f ;

[0053] Step S5, combining the screened static background feature points with the eight-point method to solve a more accurate basic matrix Fn or homography matrix Hn, repeating steps 3 and 4 until the classification of the set to which all feature points belong no longer changes, thereby obtaining the most accurate classification of static background feature points and dynamic target feature points;

[0054] Step S6, solving the affine transformation matrix according to the correspondence relationship between the static background feature points of two adjacent frames and the motion parameter model, performing background compensation on the current frame, using the inter-frame difference method to differentiate the two frames and perform binarization and morphological processing to obtain the corresponding moving target frame;

[0055] Step S61, solving the affine transformation matrix according to the correspondence relationship between the static background feature points of two adjacent frames of images and the motion parameter model, obtaining the motion parameter value, performing background compensation on the current frame according to the accurate global motion parameter, interpolating the grayscale image after background compensation using a bilinear interpolation algorithm, and performing difference processing on the two frames of images using an inter-frame difference method;

[0056] Step S62, performing adaptive threshold binarization processing and Canny edge detection on the grayscale image after inter-frame difference, and performing morphological processing on the results of the two to obtain the corresponding moving target frame;

[0057] Step S7: Use the YOLO network to detect all potential objects in the image and obtain the object target frame detected by YOLO;

[0058] Step S8, traverse the moving target frame, merge the moving target frame whose center point is located in the object target frame detected by YOLO, obtain a complete moving target frame, match the merged moving target frame information with the target frame information detected by YOLO, and the calculation formula of the overlap rate is as follows:

[0059]

[0060] In the above formula, λ represents the overlap ratio between the moving target rectangular frame Z1 detected by inter-frame difference and the target rectangular frame Z2 detected by YOLO;

[0061] Step S9: Screen out the detection objects whose overlap rate is greater than the set threshold, which are the actual moving targets in the current image.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] The method of the present invention completes the feature point classification task based on IMU-based camera motion estimation and improved maximum inter-class variance method combined with iterative calculation, and compares and matches the coincidence of YOLO target detection information with the motion information of inter-frame difference detection, thereby completing the screening and marking of the actual moving targets in the current image, thereby achieving the effect of improving the precision and accuracy of moving target recognition under dynamic background. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present disclosure and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 It is a flow chart of a method for detecting a moving target assisted by an IMU in a dynamic situation according to the present invention;

[0066] Figure 2 is an example diagram of feature point matching in an embodiment of the present invention;

[0067] Figure 3 is an epipolar drawing diagram and a characteristic point homography mapping diagram in an embodiment of the present invention;

[0068] Figure 4 is a flow chart of matching the degree of coincidence between the moving target frame information and the target frame information detected by YOLO in an embodiment of the present invention;

[0069] Figure 5 2 is an example diagram of a moving target detection process in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to facilitate those of ordinary skill in the art to understand and implement the present invention, the following detailed description is given to each step of the method proposed by the present invention, and it should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

[0071] Example

[0072] like Figure 1 As shown, the present invention discloses an IMU-assisted moving target detection method in a dynamic situation, comprising the following steps:

[0073] Step S1, using the real-time robust SURF algorithm to extract feature points from two adjacent frames of images, then using the BBF search algorithm to match the feature points, and introducing symmetry constraints to obtain robust and accurate feature point pairs;

[0074] Step S11, read in the video stream, read two adjacent frames of images in sequence for processing, first use the SURF algorithm to extract feature points from the two adjacent frames of images, and then use the BBF search algorithm to find the two closest feature points in the next frame; if the next closest distance divided by the closest distance is less than a preset threshold, it is determined to be a pair of matching points;

[0075] Step S12: using symmetric constraints to remove mismatched feature point pairs and improve the matching success rate, the feature points are screened by the bidirectional matching method. For the feature points in the previous frame, the forward matching method is used to find the matching feature points in the current frame, and then the reverse matching method is used to find the matching feature points of the current frame in the previous frame. Figure 2 As shown, an accurate pair of matching feature points is finally obtained;

[0076] Step S2, performing relative coordinate transformation on the IMU sensor data, obtaining the rotation information of the camera by integrating the angular velocity, obtaining the translation information of the camera by quadratically integrating the acceleration, and solving the corresponding basic matrix or homography matrix according to whether the camera moves between two consecutive frames;

[0077] Step S21, performing relative coordinate transformation on the IMU sensor data according to the relative position relationship between the coordinate system where the camera is located and the coordinate system where the inertial measurement unit is located. After the relative coordinate transformation on the IMU sensor data, the gyroscope and accelerometer included in the IMU respectively provide the angular velocity and acceleration of the camera, and the rotation information of the camera is obtained by integrating the angular velocity, and the translation information of the camera is obtained by quadratically integrating the acceleration;

[0078] Step S22: Calculate the transformation relationship between two consecutive frames according to the obtained camera rotation and translation information, and express the rotation information between two consecutive frames by the formula:

[0079]

[0080] The translation information between two consecutive frames is expressed as:

[0081] T=t k -t k-1 (2)

[0082] In the above formula, R represents the rotation transformation matrix from the previous frame image to the current frame image; R k-1 Represents the rotation state matrix of the previous frame image; R k represents the rotation state matrix of the current frame image; T represents the translation transformation matrix from the previous frame image to the current frame image; t k-1 Represents the translation state matrix of the previous frame image; t k Represents the translation state matrix of the current frame image;

[0083] Step S23, solve the corresponding basic matrix or homography matrix according to whether the camera moves between two consecutive image frames; if the translation distance |T| from the previous frame image to the current frame image is ≠ 0, use the IMU sensor data to calculate the initial basic matrix, and the calculation formula of the basic matrix is ​​as follows:

[0084] F=K -T [T] X R -1 (3)

[0085] In the above formula, F represents the basic matrix corresponding to the previous frame image and the current frame image, R and T are respectively the rotation matrix and translation matrix from the previous frame to the current frame obtained in step S22, and K is the built-in parameter of the camera;

[0086] If the translation distance |T| from the previous frame image to the current frame image = 0, the IMU sensor data is used to calculate the initial homography matrix. The calculation formula of the homography matrix is ​​as follows:

[0087] H=KRK -1 (4)

[0088] In the above formula, H represents the homography matrix corresponding to the previous frame image and the current frame image;

[0089] Step S3: Select the corresponding basic matrix or homography matrix as the input matrix according to whether the camera moves between two consecutive frames, and then draw the epipolar lines according to the basic matrix and the feature points, or obtain the homography projection feature points corresponding to the feature points of the current frame according to the homography matrix and the feature points. The specific example is as follows: Figure 3 As shown, Figure 3 Where (a) and (c) are Figure 3 The original images in (b) and (d) are Figure 3 (b) is the correspondence diagram between the epipolar lines and feature points; Figure 3 (d) is the correspondence diagram between the feature points of the previous frame (hollow points) and the feature points of the homography projection (solid points);

[0090] Step S31: If the translation distance |T| from the previous frame image to the current frame image is 0, the corresponding basic matrix is ​​selected as the input matrix, and the corresponding epipolar line is drawn according to the relationship between the basic matrix and the feature points, wherein the epipolar constraint formula is as follows:

[0091]

[0092] In the above formula, p k 、p k-1 Respectively represent the pixel positions of the corresponding matching feature points in the current frame and the previous frame, and F is the basic matrix before the two frames, then:

[0093] l k =Fp k-1 (6)

[0094] In the above formula, l k Represents the epipolar line corresponding to the feature point of the current frame, and has:

[0095] l k :ax+by+c=0 (7)

[0096] In the above formula, x and y are l k The pixel coordinates on the , a, b and c are the epipolar lines l k The coefficient of

[0097] Step S32: If the translation distance |T| from the previous frame image to the current frame image is ≠ 0, the epipolar constraint is invalid at this time, and the corresponding homography matrix is ​​selected as the input matrix. The homography mapping projection feature points corresponding to the feature points of the current frame are obtained according to the homography matrix and the feature points. The formula for obtaining the homography mapping projection feature points is as follows:

[0098] p' k =Hp k-1 (8)

[0099] In the above formula, p k-1 、p' kThey represent the pixel positions of the feature points of the previous frame and the feature points of the corresponding homography projection of the current frame, respectively, and H is the homography matrix before two frames;

[0100] Step S4, calculating the distance error between each feature point and the corresponding epipolar line or homography projection feature point, performing histogram quantization on the distance errors of all feature points, taking the quantized distance error as input, and obtaining the optimal threshold by the maximum inter-class variance method, and classifying the extracted feature points into two parts: static background feature points and dynamic target feature points according to the threshold;

[0101] Step S41, calculate the distance error between each feature point and the corresponding epipolar line or the projected feature point. When the input matrix is ​​the basic matrix, the distance error is the distance d from the feature point to the corresponding epipolar line:

[0102]

[0103] In the above formula, (u, v) is the coordinate of the feature point, a, b and c are the coefficients of the corresponding polar line;

[0104] When the input matrix is ​​a homography matrix, the distance error is the distance d between the original feature point and the corresponding projected feature point:

[0105]

[0106] In the above formula, (u' k ,v' k )、(u k ,v k ) are the pixel coordinates of the original feature point and the corresponding projected feature point respectively;

[0107] Step S42: quantize the distance error d of each feature point into L intervals from 0 to L. Assume that the distance error satisfies d i-1 ≤d <d i The feature points belong to the interval [i-1,i], and the corresponding number of feature points is n i Then the total number of feature points N is expressed as:

[0108] N=n1+n2+...+n k +...+n L (11)

[0109] The probability p that the number of feature points contained in each interval accounts for the total number i It is expressed as:

[0110]

[0111] Step S43: Set a threshold d k , let d min <d k<d max ; and use it to classify the feature points into a static background feature point set C b and dynamic foreground feature point set C f , where C b It consists of all feature points whose distance error is in the interval [0, k], C f It is composed of all feature points whose distance error is in the interval [k,L], respectively represented by P b and P f represents the probability of background point set and foreground point set, then C b , C f The probability calculation formula is as follows:

[0112]

[0113] Assign to C b , C f The average distance errors of the feature points of the point set are:

[0114]

[0115] Step S44: Based on the data obtained in step S43, the between-class variance It is expressed as:

[0116]

[0117] Since the two means u b and u f The farther apart they are, the greater the inter-class variance, and the distance error d traversed from 0-L i , get the maximum between-class variance

[0118]

[0119] Let the distance error with the largest inter-class variance be recorded as the optimal threshold d th ; Since the distance error of the static background feature points is smaller than the distance error of the dynamic foreground feature points, when d <d th When d>d th When it is judged as a dynamic foreground feature point, the filtered static background feature point set C is obtained b and dynamic foreground feature point set C f ;

[0120] Step S5, the screened static background feature points are combined with the eight-point method to solve a more accurate basic matrix Fn or homography matrix Hn, and steps 3 and 4 are repeated until the classification of the set to which all feature points belong no longer changes, thereby obtaining the most accurate classification of static background feature points and dynamic target feature points. The feature point classification example is as follows Figure 5 As shown in (b), Figure 5 (a) is the original image of the current frame. Figure 5 (a) Only one car on the road is moving, and all other cars in the parking lot remain stationary;

[0121] Step S6, solving the affine transformation matrix according to the correspondence relationship between the static background feature points of two adjacent frames and the motion parameter model, performing background compensation on the current frame, using the inter-frame difference method to differentiate the two frames and perform binarization and morphological processing to obtain the corresponding moving target frame;

[0122] Step S61, solve the affine transformation matrix according to the correspondence relationship between the static background feature points of two adjacent frames and the motion parameter model, obtain the motion parameter value, perform background compensation on the current frame according to the accurate global motion parameter, use the bilinear interpolation algorithm to interpolate the grayscale image after background compensation, and use the frame difference method to perform difference processing on the two frames, such as Figure 5 As shown, Figure 5 (c) is the inter-frame difference effect diagram. Figure 5 (d) is the effect of binarization processing. Figure 5 Middle (e) is the morphological processing effect diagram;

[0123] Step S62, performing adaptive threshold binarization processing and Canny edge detection on the grayscale image after inter-frame difference, and performing morphological processing on the results of the two to obtain the corresponding moving target frame;

[0124] Step S7: Use the YOLO network to detect all potential objects in the image, such as Figure 5 As shown in (f), the object target frame detected by YOLO is obtained;

[0125] Step S8, traverse the moving target frame, merge the moving target frame whose center point is located in the object target frame detected by YOLO, obtain a complete moving target frame, and match the merged moving target frame information with the target frame information detected by YOLO. The coincidence matching process is as follows: Figure 4 As shown in the figure, the calculation formula of the overlap rate is as follows:

[0126]

[0127] In the above formula, λ represents the overlap ratio between the moving target rectangular frame Z1 detected by inter-frame difference and the target rectangular frame Z2 detected by YOLO;

[0128] Step S9: Filter out the detection objects whose overlap rate is greater than the set threshold, which are the actual moving targets in the current image. The moving target detection result in this embodiment is as follows: Figure 5 As shown in (g), Figure 5(h) is the true value of the moving target.

[0129] In summary, the method of the present invention combines the camera motion estimation based on IMU and the improved maximum inter-class variance method with iterative calculation to complete the feature point classification task, and compares and matches the YOLO target detection information with the motion information of the inter-frame difference detection, so as to complete the screening and marking of the actual moving target in the current image, so as to achieve the effect of improving the precision and accuracy of moving target recognition under dynamic background.

[0130] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A method for detecting moving targets assisted by IMU in dynamic situations, characterized in that: The following steps are involved: Step S1, using the real-time robust SURF algorithm to extract feature points from two adjacent frames of images, then using the BBF search algorithm to match the feature points, and introducing symmetry constraints to obtain robust and accurate feature point pairs; Step S2, performing relative coordinate transformation on the IMU sensor data, obtaining the rotation information of the camera by integrating the angular velocity, obtaining the translation information of the camera by quadratically integrating the acceleration, and solving the corresponding basic matrix or homography matrix according to whether the camera moves between two consecutive frames; Step S3, selecting a corresponding basic matrix or homography matrix as an input matrix according to whether the camera moves between two consecutive frames, and then drawing epipolar lines according to the basic matrix and feature points, or obtaining homography mapping projection feature points corresponding to feature points of the current frame according to the homography matrix and feature points; Step S4, calculating the distance error between each feature point and the corresponding epipolar line or homography projection feature point, performing histogram quantization on the distance errors of all feature points, taking the quantized distance error as input, and obtaining the optimal threshold by the maximum inter-class variance method, and classifying the extracted feature points into two parts: static background feature points and dynamic target feature points according to the threshold; Step S5, combining the selected static background feature points with the eight-point method to solve a more accurate basic matrix Fn or homography matrix Hn, repeating steps S3 and S4 until the classification of the set to which all feature points belong no longer changes, thereby obtaining the most accurate classification of static background feature points and dynamic target feature points; Step S6, solving the affine transformation matrix according to the correspondence relationship between the static background feature points of two adjacent frames and the motion parameter model, performing background compensation on the current frame, using the inter-frame difference method to differentiate the two frames and perform binarization and morphological processing to obtain the corresponding moving target frame; Step S7: Use the YOLO network to detect all potential objects in the image and obtain the object target frame detected by YOLO; Step S8, traverse the moving target frame, merge the moving target frame whose center point is located in the object target frame detected by YOLO, obtain a complete moving target frame, and match the merged moving target frame information with the target frame information detected by YOLO; Step S9: Screen out the detection objects whose overlap rate is greater than the set threshold, which are the actual moving targets in the current image.

2. The IMU-assisted moving target detection method in a dynamic situation according to claim 1, characterized in that: The specific contents of step 1 are as follows: Step S11, read in the video stream, read two adjacent frames of images in sequence for processing, first use the SURF algorithm to extract feature points from the two adjacent frames of images, and then use the BBF search algorithm to find the two closest feature points in the next frame; if the next closest distance divided by the closest distance is less than a preset threshold, it is determined to be a pair of matching points; Step S12, using symmetric constraints to remove mismatched feature point pairs and improve the matching success rate, the feature points are screened through the two-way matching method, and for the feature points in the previous frame, the forward matching method is used to find the matching feature points in the current frame, and then the reverse matching method is used to find the matching feature points of the current frame in the previous frame, and finally an accurate matching feature point pair is obtained.

3. The IMU-assisted moving target detection method in a dynamic situation according to claim 1, characterized in that: The specific content of step S2 is as follows: Step S21, performing relative coordinate transformation on the IMU sensor data according to the relative position relationship between the coordinate system where the camera is located and the coordinate system where the inertial measurement unit is located. After the relative coordinate transformation on the IMU sensor data, the gyroscope and accelerometer included in the IMU respectively provide the angular velocity and acceleration of the camera, and the rotation information of the camera is obtained by integrating the angular velocity, and the translation information of the camera is obtained by quadratically integrating the acceleration; Step S22: Calculate the transformation relationship between two consecutive frames according to the obtained camera rotation and translation information, and express the rotation information between two consecutive frames by the formula: The translation information between two consecutive frames is expressed as: T=t k -t k-1 (2) In the above formula, R represents the rotation transformation matrix from the previous frame image to the current frame image; R k-1 Represents the rotation state matrix of the previous frame image; R k represents the rotation state matrix of the current frame image; T represents the translation transformation matrix from the previous frame image to the current frame image; t k-1 Represents the translation state matrix of the previous frame image; t k Represents the translation state matrix of the current frame image; Step S23, solving the corresponding basic matrix or homography matrix according to whether the camera moves between two consecutive image frames; If the translation distance |T| from the previous frame image to the current frame image is ≠ 0, the IMU sensor data is used to calculate the initial basic matrix. The calculation formula of the basic matrix is ​​as follows: F=K -T [T] X RK -1 (3) In the above formula, F represents the basic matrix corresponding to the previous frame image and the current frame image, R and T are respectively the rotation matrix and translation matrix from the previous frame to the current frame obtained in step S22, and K is the built-in parameter of the camera; If the translation distance |T| from the previous frame image to the current frame image = 0, the IMU sensor data is used to calculate the initial homography matrix. The calculation formula of the homography matrix is ​​as follows: H=KRK -1 (4) In the above formula, H represents the homography matrix corresponding to the previous frame image and the current frame image.

4. The IMU-assisted moving target detection method in a dynamic situation according to claim 1, characterized in that: The specific content of step S3 is as follows: Step S31: If the translation distance |T| from the previous frame image to the current frame image is 0, the corresponding basic matrix is ​​selected as the input matrix, and the corresponding epipolar line is drawn according to the relationship between the basic matrix and the feature points, wherein the epipolar constraint formula is as follows: In the above formula, p k 、p k-1 Respectively represent the pixel positions of the corresponding matching feature points in the current frame and the previous frame, and F is the basic matrix before the two frames, then: l k =Fp k-1 (6) In the above formula, l k Represents the epipolar line corresponding to the feature point of the current frame, and has: l k :ax+by+c=0 (7) In the above formula, x and y are l k The pixel coordinates on the , a, b and c are the epipolar lines l k The coefficient of Step S32: If the translation distance |T| from the previous frame image to the current frame image is ≠ 0, the epipolar constraint is invalid at this time, and the corresponding homography matrix is ​​selected as the input matrix. The homography mapping projection feature points corresponding to the feature points of the current frame are obtained according to the homography matrix and the feature points. The formula for obtaining the homography mapping projection feature points is as follows: p′ k =Hp k-1 (8) In the above formula, p k-1 、p' k They represent the pixel positions of the feature points of the previous frame and the corresponding homography projection feature points of the current frame, respectively, and H is the homography matrix before two frames.

5. The IMU-assisted moving target detection method in a dynamic situation according to claim 1, characterized in that: The specific content of step S4 is as follows: Step S41, calculate the distance error between each feature point and the corresponding epipolar line or the projected feature point. When the input matrix is ​​the basic matrix, the distance error is the distance d from the feature point to the corresponding epipolar line: In the above formula, (u, v) is the coordinate of the feature point, a, b and c are the coefficients of the corresponding polar line; When the input matrix is ​​a homography matrix, the distance error is the distance d between the original feature point and the corresponding projected feature point: In the above formula, (u'k,v'k), (u k ,v k ) are the pixel coordinates of the original feature point and the corresponding projected feature point respectively; Step S42: quantize the distance error d of each feature point into L intervals from 0 to L. Assume that the distance error satisfies d i-1 ≤d <d i The feature points belong to the interval [i-1,i], and the corresponding number of feature points is n i Then the total number of all feature points N is expressed as: N=n1+n2+...+n k +...+n L (11) The probability p that the number of feature points contained in each interval accounts for the total number i It is expressed as: Step S43: Set a threshold d k , let d min <d k <d max ; and use it to classify the feature points into a static background feature point set C b and dynamic foreground feature point set C f , where C b It consists of all feature points whose distance error is in the interval [0, k], C f It is composed of all feature points whose distance error is in the interval [k,L], respectively represented by P b and P f represents the probability of background point set and foreground point set, then C b , C f The probability calculation formula is as follows: Assign to C b , C f The average distance errors of the feature points of the point set are: Step S44: Based on the data obtained in step S43, the between-class variance It is expressed as: Since the two means u b and u f The farther apart they are, the greater the inter-class variance, and the distance error d traversed from 0-L i , get the maximum between-class variance Let the distance error with the largest inter-class variance be recorded as the optimal threshold d th ; Since the distance error of the static background feature points is smaller than the distance error of the dynamic foreground feature points, when d <d th When d>d th When it is judged as a dynamic foreground feature point, the filtered static background feature point set C is obtained b and dynamic foreground feature point set C f .

6. The IMU-assisted moving target detection method in a dynamic situation according to claim 1, characterized in that: The specific content of step S6 is as follows: Step S61, solving the affine transformation matrix according to the correspondence relationship between the static background feature points of two adjacent frames of images and the motion parameter model, obtaining the motion parameter value, performing background compensation on the current frame according to the accurate global motion parameter, interpolating the grayscale image after background compensation using a bilinear interpolation algorithm, and performing difference processing on the two frames of images using an inter-frame difference method; Step S62: perform adaptive threshold binarization processing and Canny edge detection on the grayscale image after inter-frame difference, and perform morphological processing on the results of the two to obtain the corresponding moving target frame.

7. The IMU-assisted moving target detection method in a dynamic situation according to claim 1, characterized in that: In step S8, the merged moving target frame information is matched with the target frame information detected by YOLO, and the calculation formula of the overlap rate is as follows: In the above formula, λ represents the overlap ratio between the moving target rectangular frame Z1 detected by inter-frame difference and the target rectangular frame Z2 detected by YOLO.

Images